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About this practice guide 

The Institute of Education Sciences (lES) publishes practice guides in education to provide 
educators with the best available evidence and expertise on current challenges in education. The 
What Works Clearinghouse (WWC) develops practice guides in conjunction with an expert panel, 
combining the panel’s expertise with the findings of existing rigorous research to produce specific 
recommendations for addressing these challenges. The WWC and the panel rate the strength 
of the research evidence supporting each of their recommendations. See Appendix A for a full 
description of practice guides. 

The goal of this practice guide is to offer specific, evidence-based recommendations for college 
and university faculty, administrators, and advisors working to improve the success of students 
academically underprepared for college. Each recommendation includes an overview of the 
practice, a summary of evidence used in support of the evidence rating, guidance on how to 
carry out the recommendation, and suggested approaches to overcome potential roadblocks. 
Each recommendation includes an implementation checklist, as guidance forgetting started with 
implementing the recommendation. 

Practice guides published by lES are available on the WWC website at http://whatworks.ed.gov 

This report was prepared for the National Center for Education Evaluation and Regional Assistance, 
Institute of Education Sciences, under the WWC contract to Development Services Croup, Inc. 
(Contract ED-IES-12-C-0084). 

Disclaimer 

The opinions and positions expressed in this practice guide are those of the authors and do not 
necessarily represent the opinions and positions of the lES or the U.S. Department of Education. 
Educators, colleges, universities, and organizations should review the practice guide and apply its 
recommendations according to their specific needs and goals. It is important to recognize that the 
practice guide represents the Judgments of the review panel regarding what constitutes sensible 
practice, based on the research that was available within the publication timeframe established in 
the review protocol. This practice guide should be used as a tool to assist in decisionmaking rather 
than as a "cookbook" or "recipe" that contains instructions to be strictly followed. Any references 
within the document to specific education products are illustrative and do not imply endorsement 
of these products to the exclusion of other products that are not referenced. 
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Introduction 


Introduction to the Practice Guide on Strategies 
for Students in Developmental Education 

A pproximately one-third of students entering higher education do so without the necessary 
skills in at least one subject to enroll and succeed in college-level work.' Underprepared col- 
lege students are less likely than their peers to complete their educational objectives. As an exam- 
ple, 1 992’s high school graduates who first entered community college and enrolled in at least one 
developmental education course had lower degree attainment rates 8 years after high school than 
first-time community college entrants who did not enroll in any developmental education courses 
(28% vs. 36%). For students first entering 4-year institutions, the picture was even bleaker, with stu 
dents enrolling in at least one developmental course earning a degree at a much lower rate (52%) 
than first-time entrants to 4-year institutions not enrolling in developmental courses (78%). ^ 


Colleges and universities have developed 
and adopted various strategies to help under- 
prepared students acquire the skills and 
knowledge needed to move into college-level 
courses.^ These policies and practices have 
been referred to as “developmental educa- 
tion,” “remedial education,” or “college-readi- 
ness programs.”^ While all of these labels are 
currently used in postsecondary education 
and have been debated, state policymakers, 
higher education administrators, and faculty 
recognize that these policies and strategies 
can help create conditions that give underpre- 
pared students the best chance of realizing 
their educational goals. 

This practice guide brings together two 
distinct sets of evidence-based practices: 
those that focus on instructional interven- 
tions, and those that focus on structural 
interventions. More specifically, two of the six 
recommended practices focus on student and 
faculty interactions in the classroom. These 
interventions involve curricular decisions that 
affect the content and pacing of classroom 
instruction. The second set includes four 
practical recommendations focused on struc- 
ture and largely involve systems for determin- 
ing which types of supports will be offered 
or mandated for students in, or at risk of 
requiring, developmental education. Together, 
these six practices can help create a learning 
environment that maximizes the chances that 
students will succeed in their coursework 
and chosen degree program. These practices 
can be implemented widely, regardless of 


existing policies on developmental or reme- 
dial education. However, without the support 
of policymakers and administrators, they will 
be difficult to implement at the institutional 
level. 

Audience and institutions of 
interest 

Administrators and faculty at institutions of 
higher education should find these practi- 
cal recommendations helpful in meeting the 
needs of students who are underprepared 
for college-level work. Audiences that may 
be particularly interested in the instructional 
practices include program coordinators or 
directors of developmental education, cur- 
riculum design professionals, and faculty 
who teach developmental education. Certain 
interest groups, such as philanthropic leaders 
and associations engaged in developmental 
education reform, will find the evidence 
regarding instructional interventions for 
developmental education useful. Academic 
leaders in state and community colleges will 
be interested in interventions that mitigate 
the impact of lack of readiness for college- 
level coursework. 

Audiences particularly interested in the 
structural interventions may include higher 
education administrators — especially those in 
financial aid offices, student counseling divi- 
sions, academic advisors, admissions offices. 
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articulation officers at community colleges 
and universities, and liaisons who collaborate 
with high school systems to foster college 
readiness. Administrative leaders — including 
presidents, deans, provosts, and depart- 
ment chairs — can find recommendations that 
shape practices at the institutional, college, 
or departmental level. Test developers may 
find the information on assessment and 
placement helpful as they work to create new 
assessments and refine existing ones. 

While the majority of students in need of 
additional developmental education in higher 
education are enrolled in community col- 
leges, four-year institutions also serve stu- 
dents with developmental education needs. ^ 
Even though the majority of the research with 
students in developmental education has 
been conducted in community colleges, this 
guide includes evidence conducted at four- 
year institutions that met our review criteria. 
Faculty, institutional leaders, and advisors 
from both sectors are encouraged to extrapo- 
late from the recommendations and evidence 
presented for their own institutional contexts. 
The intended audience for this practice guide 
is all postsecondary institutions that serve 
students who are placed in, have been recom- 
mended to, or are at-risk of being placed into 
developmental education. 

Scope of the practice guide 

The goal of this practice guide is to pres- 
ent rigorously and systematically reviewed 
evidence in support of practical recommenda- 
tions for improving outcomes for students 
at-risk for, or placed into, developmental 
education. The practice guide primarily 
focuses on interventions or practices aimed 
at improving students’ progress through 
developmental education, credit accumula- 
tion and persistence, academic achievement, 
and degree attainment. 

To determine the scope of the practice guide, 
expert researchers and practitioners initially 
considered more than twenty practices, and 


narrowed the focus to eight practice areas, 
based on available evidence and expert 
opinion. The eight practices identified were: 
1) comprehensive and integrated support 
programs; 2) contextualized instruction; 3) 
early assessment programs for at-risk high 
school students; 4) enhanced and early 
alert advising; 5) performance-based mon- 
etary incentives; 6) practices to accelerate, 
compress, or mainstream developmental 
education; 7) practices to modify information 
used to make placement decisions; and 8) 
practices to teach metacognition, productive 
persistence, and college success skills. This 
introduction includes a description of how 
the assembled team of expert researchers 
and practitioners reviewed the evidence on 
these practices to develop and refine their 
final recommendations. 

Based upon review of the evidence, the 
expert panel decided not to include practice 
recommendations for early assessment pro- 
grams for at-risk high school students, and 
contextualized instruction (without compres- 
sion or acceleration) in the practice guide. 

The expert panel also found little evidence to 
support broad practices to teach metacogni- 
tion, productive persistence, and college suc- 
cess skills. With evidence on positive effects 
of teaching self-regulated learning to stu- 
dents in developmental education, the expert 
panel refined this practice recommendation 
to teaching self-regulated learning. 

Even though the panel did not include early 
assessment programs for at-risk high school 
students as one of its recommendations, one 
of the primary ways to help students suc- 
ceed in higher education is engage students 
in early assessment and implement college- 
readiness interventions, so that they do not 
need developmental education courses in 
the first place. Several states have initiated 
programs to assess students during their 
junior year of high school and offer courses 
for underprepared students during their 
senior year.® These programs involve mea- 
suring high school students’ readiness for 
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college-level coursework, which can inform 
students and their families about the need 
for skill building in math, reading, or writing, 
and help them avoid formal placement in 
developmental education.^ While there are 
few studies of these relatively new programs, 
the expert panel convened by the What 
Works Clearinghouse (WWC) to review the 
evidence of these interventions believes that 
these early assessment and college-readiness 
interventions show promise, especially when 
implemented statewide. Due to the limited 
available evidence, however, early assess- 
ment and college-readiness interventions are 
not included as one of the six recommenda- 
tions in this practice guide. 

Once students arrive on a college campus, a 
primary challenge that practitioners face in 
improving outcomes for underprepared stu- 
dents is identifying those students who may 
need extra assistance to be college-ready. 
Unfortunately, many policies divide students 
into two broad categories: students who are 
ready for college-level work and those who 
are not.® Within those two broad categories, 
states and postsecondary institutions have 
established multiple levels of developmental 
education progression, and college-level 
coursework (typically denoted in 100-level, 
200-level, etc. courses). Since readiness for 
college-level work exists on a continuum, 
with most students displaying different 
degrees of readiness for different subjects, 
the expert panel recommends tailoring inter- 
ventions for students at different degrees of 
readiness for college-level work. 

A second challenge is that there is no single 
“magic bullet.” Many interventions may be 
more effective in combination than if imple- 
mented alone. For example, the evidence 
suggests that the bundling of interventions 
is associated with relatively strong effects 
in developmental students’ outcomes.® 

These bundles of interventions can include 
some combination of full-time enrollment, 
enhanced advising, tutoring, accelerated 


coursework, a cohort model, or student finan- 
cial incentives, among others. 

Finally, the quality of instruction that occurs 
in developmental education likely plays a 
large role in students’ eventual success in 
postsecondary education. While this guide 
includes evidence-based teaching practices, 
such as self-regulated learning, there are 
other instructional practices not included that 
likely would also improve learning and result 
in higher rates of success for developmen- 
tal students. Evaluations of many of these 
practices fell outside the scope of this guide, 
often because their efficacy has not been 
assessed rigorously using developmental 
students. 

Summary of recommendations 

1. Recommendation 1: Use multiple mea- 
sures to assess postsecondary readiness 
and place students. 

2. Recommendation 2: Require or incentivize 
regular participation in enhanced advising 
activities. 

3. Recommendation 3: Offer students perfor- 
mance-based monetary incentives. 

4. Recommendation 4: Compress or main- 
stream developmental education with 
course redesign. 

5. Recommendation 5: Teach students how 
to become self-regulated learners. 

6. Recommendation 6: Implement com- 
prehensive, integrated, and long-lasting 
support programs. 

Development of the practice guide 

An expert panel guided the development of 
the practice guide. The panel included: Dr. 
Thomas Bailey, director. Community Col- 
lege Research Center and Ceorge and Abby 
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O'Neill Professor of Economics and Education, 
Teachers College, Columbia University; Dr. 
Joanne Bashford, president, InterAmerican 
Campus, Miami Dade College; Dr. Angela 
Boatman, assistant professor of Public Policy 
and Higher Education, Vanderbilt University; 
John Squires, director of High School to Col- 
lege Readiness, Southern Regional Education 
Board; and Dr. Michael Weiss, senior associ- 
ate, MDRC. The panel represented perspec- 
tives of practitioner leaders working on 
reforms in developmental education and the 
expertise of senior scholars in this field. 

Once the expert panel was confirmed, 
practice guide development began with the 
Panel’s identification of 21 initial practices 
to improve outcomes for students in devel- 
opmental education, in collaboration with 
the senior content expert (a member of the 
staff leadership team). In the first meeting, 
panelists received a briefing book on these 
21 collaboratively-identified practices for 
supporting the success of underprepared 
students. The briefing book provided practice 
descriptions, contexts of implementation for 
the practice, and brief summaries of evidence 
for 21 interventions targeted at improving 
outcomes for students placed into, and at-risk 
of placement in, developmental education. 
Based on discussion of briefing materials on 
these 21 practices, the expert panel agreed 
upon an initial set of recommended prac- 
tices for systematic search and review of the 
evidence. The initial list of practices provided 
a beginning foundation to define the scope 
of the practice guide. In their deliberations, 
panel members raised discussion about other 
practices, and freely deliberated as they nar- 
rowed a final list of practices for systematic 
search and review of evidence. 

Panel members were asked to consider three 
questions about each possible practice: 

1 . In your professional opinion, is this prac- 
tice likely to help underprepared students 
succeed? In short, does it work? 


2. Is there likely to be a substantial evidence 
base to support the effectiveness of this 
practice? 

3. Can this practice be replicated in a wide 
variety of locales, or is it highly specific 
to the institution or setting in which it has 
been developed? 

The result of this meeting was a set of prac- 
tices that panel members believed were 
promising enough to warrant a close inves- 
tigation. The WWC developed a protocol that 
described how the review of these practices 
would be carried out, and this protocol 
included a systematic literature search strat- 
egy for each recommended practice. From 
this systematic search of literature reported 
between January 1 , 1 995, and August 31 , 

201 5, 25,697 studies were identified from 
abstracts in the EBSCO and ProQuest research 
databases, and 95 additional studies were 
identified from supplemental searches of 
relevant websites, literature reviews, and 
reference harvesting from other studies. 

Based upon the review protocol, studies were 
screened for relevance (e.g., whether they 
investigated the effects of an intervention 
and whether the sample consisted exclusively 
or largely of students in developmental edu- 
cation), and the full text of each study Judged 
to be possibly relevant (based on information 
in the title and abstract) was obtained. 

Trained and certified reviewers then exam- 
ined these studies against the standards of 
the WWC, using the What Works Clearing- 
house Procedures and Standards Handbook, 
Version 3.0. Among the 48 eligible reports 
and publications, 10 studies met WWC stan- 
dards without reservations or met WWC 
standards with reservations. 

The panel then met to discuss the results of 
the literature search. Based on the findings 
and review of the evidence, the panel then 
made its final determination regarding the 
level of evidence for each practice. The panel 
was supported in its work by the reviews 
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conducted by WWC staff, but the panel mem- 
bers made the final determination for each 
type of evidence. 

In making its determination, the panel 
used the levels of evidence criteria, which 
are set by WWC policy, in Table A in this 
Introduction. 

The full text of 439 studies was screened for 
relevance. While 68 studies met criteria for 


review in eight original practices, 48 studies 
were found to be eligible for review under 
the protocol, and aligned with the final six 
practice recommendations. As part of their 
deliberations, the panel considered evidence 
from 20 studies which were originally rated 
in the areas of early assessment and interven- 
tion, contextualized instruction, and meta- 
cognition. After review of the evidence, these 
20 studies no longer aligned with the panel’s 
recommendations. 
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Figure A. Prisma flow diagram for study identification, screening, eligibility, and inclusion status/ratings 


Records identified through database 
searching 

(eight original practice guide areas; six final 
eligible practices)* 

(r7 = 25,697) 


Additional records identified 
through other sources 
(n = 95) 


Total records identified 
(r7 = 25,792) 


Screened for relevance 
(n = 439) 

i 

Eligible for review 
(Six final eligible practices) 

{n = 48) 

1 . Studies that met WWC Standards 
with OR without reservations 
(n= 10) 

2. Eligible studies that did not meet 
WWC standards, used as credible 
evidence 

(«=5) 

3. Studies with high-quality correla- 
tional designs, used as evidence 
(n = 2) 

4. Studies with samples of equal 
or greater than 40% students in 
developmental education, and less 
than 66%, that met WWC standards, 
used as evidence 

in = 2) 





Records excluded 
(r7 = 25,353) 


Full-text articles excluded 
(r7= 391) 


(Among six final eligible practices) 
Eligible studies not meeting WWC 
standards and not used as evidence 
in = 36) 

Effectiveness not soieiy due to 
intervention (n = 3) 

Equivaience not demonstrated (n = 30) 
Did not meet regression discontinuity 
standards (n = 2) 

Outcomes did not meet review 
requirements (n = 1) 


Total = 19 studies used as 
evidence to support ratings 


* Accelerated Instruction; Advising: Incentives; Integrated, Comprehensive, & Long-Lasting Supports; Multiple 
Measures: Self-Regulated Learning 

(Three practices were excluded for lack of evidence: Contextualized Instruction, Early Assessment & Interven- 
tion, Metacognition.) 
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Table A. Institute of Education Sciences levels of evidence for WWC practice guides 


Criteria 


STRONG 
Evidence Base 


MODERATE 
Evidence Base 


MINIMAL 
Evidence Base 


Validity 

High internal 
validity (high- 
quality causal 
designs). Studies 
must meet WWC 
design standards 
with or without 
reservations." 

AND 

High external 
validity (requires 
multiple studies 
with high-quality 
causal designs 
that represent 
the population on 
which the recom- 
mendation is 
focused). Studies 
must meet WWC 
design standards 
with or without 
reservations. 

High internal valid- 
ity but moderate 
external validity (i.e., 
studies that support 
strong causal conclu- 
sions but generaliza- 
tion is uncertain). 

OR 

High external validity 
but moderate internal 
validity (i.e., stud- 
ies that support the 
generality of a rela- 
tion but the causality 
is uncertain).’^ 

The research may include 
evidence from studies that 
do not meet the criteria 
for moderate or strong 
evidence (e.g., case studies, 
qualitative research). 

Effects on relevant 

outcomes 

Consistent posi- 
tive effects with- 
out contradictory 
evidence (i.e., no 
statistically sig- 
nificant negative 
effects) in studies 
with high internal 
validity. 

A preponderance of 
evidence of positive 
effects. Contradic- 
tory evidence (i.e., 
statistically signifi- 
cant negative effects) 
must be discussed 
by the panel and 
considered with 
regard to relevance 
to the scope of the 
guide and intensity 
of the recommenda- 
tion as a component 
of the intervention 
evaluated. 

There may be weak or 
contradictory evidence of 
effects. 
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Criteria 


STRONG 
Evidence Base 


MODERATE 
Evidence Base 


MINIMAL 
Evidence Base 


Relevance to scope 


Direct relevance to 
scope (i.e., ecological 
validity) — relevant 
context (e.g., class- 
room vs. laboratory), 
sample (e.g., age and 
characteristics), and 
outcomes evaluated. 


Relevance to scope 
(ecological validity) 
may vary, including 
relevant context 
(e.g., classroom vs. 
laboratory), sample 
(e.g., age and char- 
acteristics), and out- 
comes evaluated. At 
least some research 
is directly relevant 
to scope (but the 
research that is 
relevant to scope 
does not qualify as 
strong with respect 
to validity). 


The research may be 
out of the scope of the 
practice guide. 


Relationship between 
research and 
recommendations 


Panel confidence 


Direct test of the rec- 
ommendation in the 
studies or the recom- 
mendation is a major 
component of the 
intervention tested in 
the studies. 


The panel has a high 
degree of confidence 
that this practice is 
effective. 


Intensity of the 
recommendation as 
a component of the 
interventions evalu- 
ated in the studies 
may vary. 


The panel deter- 
mines that the 
research does not 
rise to the level of 
strong but is more 
compelling than 
a minimal level of 
evidence. 

The panel may 
not be confident 
about whether the 
research has effec- 
tively controlled for 
other explanations 
or whether the 
practice would be 
effective in most or 
all contexts. 


Studies for which the 
intensity of the recom- 
mendation as a compo- 
nent of the interventions 
evaluated in the studies 
is low, and/or the rec- 
ommendation reflects 
expert opinion based on 
reasonable extrapola- 
tions from research. 

In the panel’s opinion, 
the recommendation 
must be addressed as 
part of the practice 
guide; however, the 
panel cannot point to 
a body of research that 
rises to the level of mod- 
erate or strong. 
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Criteria 

STRONG 

MODERATE 

MINIMAL 

Evidence Base 

Evidence Base 

Evidence Base 

Roie of expert 
opinion 

Not applicable 

Not applicable 

Expert opinion based 
on defensible interpreta- 


tions of theory (theories). 
(In some cases, this 
simply means that the 
recommended practices 
would be difficult to 
study in a rigorous, 
experimental fashion; 
in other cases, it means 
that researchers have not 
yet studied this practice.) 

When assessment For assessments. For assessments, evi- Not applicable 
is the focus of the meets the standards dence of reliability that 
recommendation of The Standards for meets The Standards for 

Educational and Psy- Educational and Psycho- 
chological Testing.'^ logical Testing but with 

evidence of validity from 
samples not adequately 
representative of the 
population on which 
the recommendation is 
focused. 
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Table B summarizes the recommended levels of evidence for each practice. The expert 
panel assessed the levels of evidence based on the lES criteria outlined in Table A. 


TABLE B. LEVELS OF EVIDENCE FOR EACH PRACTICE RECOMMENDATION. 


PRACTICE RECOMMENDATION STRONG 

MODERATE 

MINIMAL 

1. Use multiple measures to assess 

postsecondary readiness and place students. 


• 

2. Require or incentivize regular participation in 
enhanced advising activities 

• 


3. Offer students performance-based monetary 
incentives. 

• 


4. Compress or mainstream developmental 
education with course redesign. 


• 

5. Teach students how to become self-regulated 
learners. 


• 

6. Implement comprehensive, integrated, and 
long-lasting support programs. 

• 



Criteria for review of the evidence 

The review protocol for the practice guide 
provides detail on the product scope, study 
review criteria, procedures for systematic 
search, and methods used for study review. 
For example, the review protocol for the 
practice guide established criteria regarding 
the study populations that would be 
considered eligible for review. In order for a 
study to be eligible for review, and used for 
a strong, moderate, or minimal evidence 
rating, a study must have included 
postsecondary students in the United States 
or Canada (including students who have 
not yet started their college careers), at 
least two-thirds of whom are in, have been 
recommended for, or are at risk for being 
placed into developmental education. 

To be eligible for review and used for a 

moderate or minimal evidence rating. 


a study must have included postsecondary 
students in the United States or Canada 
(including students who have not yet 
started their college careers), at least 
40 percent of whom are in, have been 
recommended for, or are at risk for being 
placed into developmental education. 

The review protocol also established outcomes 
relevant to the review, briefly defined below. 
Within each outcome area, study reviewers 
follow guidance regarding which outcome 
measures are to be considered primary, 
and which are considered supplemental. 

■ College access and enrollment refers 
to the process of applying to, actually 
enrolling in, and attending a postsecondary 
institution. Examples of ways that 
enrollment might be operationally defined 
in studies include (a) applied vs. did not 
apply to college, (b) number of applications 
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Study Eligibility Criteria 
(see review protocol) 

Publication Timeframe: Studies reported between January 1, 1995, and August 31, 

2015 

Study Sample Location: Participants must be based in the United States or Canada. 

Sample Requirements: 

■ For use with a strong, moderate, or minimal evidence rating - At least two-thirds 
of the study sample must be students who are placed in, have been recommended 
for, or are at-risk for being placed into developmental education. 

■ For use with a moderate or minimal evidence rating - At least 40% of the study 
sample must be students who are placed in, have been recommended for, or are 
at-risk for being placed into developmental education. 

Setting: Study samples in any postsecondary education setting that meet the above 

criteria (i.e., may be two-year or four-year institutions) 


completed, (c) attended vs. did not 
attend a postsecondary institution or 
enrolled vs. not enrolled in the first 
semester or year of college, (d) selectivity 
of enrollment institution, (e) full time 
vs. part time enrollment, and (f) 4-year 
college or institution vs. 2-year college 
or institution vs. non-enrollment. Actual 
enrollment is the preferred measure. 

■ Progress through developmental 
education refers to the process of 
completing required developmental 
coursework. Examples of ways that 
progress through developmental 
education might be operationally defined 
include a) completed versus did not 
complete developmental education 
coursework, b) completed versus did 
not complete first college-level course 
in which remediation was needed, and 
c) grades earned in developmental 
courses. Passing college-level courses 
in the area of required developmental 
education is the preferred measure. 


As mentioned under “outcome period 
relevance,” the primary focus for this 
outcome domain is upon the longest 
time period observed for the outcome (or 
outcomes) in this domain. Examples of 
ways that credit accumulation might be 
operationally defined in studies include a) 
number of credits earned toward degree 
completion, b) proportion of degree- 
bearing versus non-degree-bearing 
credits earned, c) ratio of credits earned 
to credits attempted, and d) enrollment 
persistence. If a study assesses credit 
accumulation and enrollment persistence, 
the former is the preferred measure. 

■ Academic achievement refers to 
measures which assess the extent to 
which students adequately complete 
expected coursework. Examples of 
ways that academic achievement 
might be operationally defined in 
studies include grade-point average, 
departmental final exams, and the ratio 
of courses passed versus failed. 


Credit accumuiation and persistence 

refers to progress toward the completion 
of a degree, certificate, or program. 
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refers to students’ transition to a 
bachelor’s degree granting program 
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or institution, typically from a 2-year 
postsecondary institution that does 
not have any baccalaureate-degree 
granting programs (or may have 
few, selected ones). National Student 
Clearinghouse records are the preferred 
data source (which may be integrated 
into state longitudinal data systems). 

■ Attainment refers to the completion 
of a degree, certificate, or program. 
Examples of ways attainment might 
be operationally defined in a study 
include certificate completion rates and 
degree completion rates. Official school 
records are the preferred data source. 

■ Labor market refers to outcomes related 
to employment after the postsecondary 
experience. Examples of ways that labor 
market outcomes might be operationally 
defined in studies include (a) employed vs. 
not, (b) employed full-time vs. employed 
part-time, (c) employed in field of study vs. 
not, and (d) income earned. Continuous 
measures of actual income are preferred. 


How to use this practice guide 

Each section of this guide contains a recom- 
mended practice in which the practice itself 
and examples of the practice in action are 
described. The evidence base for the practice is 
reviewed, and recommendations for effectively 
implementing the practice are listed. When 
recommendations have two or more studies 
of similar interventions that assess effects 
on outcomes in the same outcome domain, 
meta-analyses were conducted (see the WWC’s 
Procedures and Standards Handbook, version 
3.0, for more information). Individual effect 
sizes for studies used as evidence in support 
of a recommendation are also presented in 
Appendix D. 

Note that this practice guide does not pro- 
vide step-by-step instructions on how to fully 
implement each practice, but does provide 
preliminary guidance on how to implement the 
recommendation, as well as common road- 
blocks and suggested approaches for overcom- 
ing them. 

Leaders on college campuses are subject to 
a host of constraints that could make imple- 
menting some of these recommendations 
difficult. These constraints come from budgets, 
available staff resources, or from system or 
state policies. With the exception of Recom- 
mendation 6, the practices identified here can 
be implemented in isolation, working within 
the constraints for a given institution. Recom- 
mendations 1-5 do not depend in a cumulative 
sense on having implemented another practice 
beforehand. In contrast, implementing compre- 
hensive supports does imply a set of interde- 
pendent policies and practices which must be 
put in place at the same time. 
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Recommendation 1: Use multiple measures to assess 
postsecondary readiness and place students. 

Most open-access institutions require incoming students to take brief standardized assessments in 
math, reading, and writing. The results of these assessments are used to place students in either 
developmental or college-level courses. However, few would argue that a single placement test is a 
perfect measure of college readiness. In fact, several studies have highlighted concerns about mis- 
placement rates.’"' For example, Scott-Clayton (2012) found that 24%-33% of students were misplaced 
when they matriculated. It was more common for students to be placed in developmental courses 
when they could have succeeded in college-level courses than the reverse. Since nearly 40% of com- 
munity college students enrolled in developmental education fail to progress to credit-bearing course- 
work,'^ misplacement into developmental education is particularly troubling. 


A basic principle of psychological measure- 
ment is that when a construct like college 
readiness is measured imperfectly, one way to 
improve measurement is to assess the con- 
struct in multiple ways. Therefore, one way 
to improve the measurement of college readi- 
ness (and therefore to reduce misplacement) 
is to use multiple measures — such as high 
school CPA, the number of years since high 
school graduation or equivalent, the number 
of courses taken in the subject (e.g., English or 
math), and the highest level taken in the sub- 
ject (e.g.. Algebra I or Algebra II) — to inform 
placement decisions. 

Figure l.l. illustrates types of measures to 
assess postsecondary readiness and place 
students. In this regard, it is notable that the 
state of California mandates the use of mul- 
tiple measures for placement into developmen- 
tal education (Title 5 §55502, California Code 
of Regulations),'^ and companies that produce 
the placement tests recognize that adding 
information beyond the test itself can improve 
placement decisions.''^ 

Summary of evidence: Minimal 
Evidence 

Experimental studies of the use of multiple 
measures for course placement investigate 
the effectiveness of different placement cri- 
teria (e.g., placement test alone versus place- 
ment test plus high school CPA) on students’ 
outcomes (such as credit accumulation and 
grades). 


The WWC identified two studies that investi- 
gated the effectiveness of alternative or mul- 
tiple measures for placement.’^ Neither study 
met WWC group design standards. One study 
was a randomized controlled trial'® for which 
the WWC could assess neither how much 
attrition the study experienced nor the extent 
to which the groups were similar at baseline. 
Similarly, in another study,^°the WWC was 
unable to assess the extent to which the 
groups were similar at baseline. Nevertheless, 
the panel believes that these studies offer 
useful insight into this recommendation. The 
studies are further discussed in Appendix D 
Table 1.1. 

Two additional non-experimental studies are 
highlighted in Appendix D Table 1.2.^' Both 
of these studies suggest that adding informa- 
tion about student high school experiences 
(in particular, high school CPA) might help 
improve placement decisions. In addition 
to multiple measures, these studies offer 
some evidence that alternative measures of 
high school preparation (compared to scores 
on the institution’s standardized place- 
ment test) can also be effective in reducing 
misplacement. 

How to carry out the 
recommendation 

Several prominent groups have provided 
guidance on using multiple measures to place 
students in developmental education. Below 


( 13 ) 


Recommendation 1 (continued) 


Figure 1.1. Types of measures to assess college readiness and place students into courses 


TYPES OF MEASURES TO ASSESS COLLEGE READINESS 
AND PLACE STUDENTS INTO COURSES 


High school grade 
point average 


Letter Grade CPA 


Score on an approved 
college readiness test 


1 o o o o o 

2 o o o o o 

3 o e o o o 

4 o o o o o 


A 

B 

C 

4.0 

3.0 

2.0 


D 

1.0 


F 

0 

Completion of, or grade 



in, a specified high school 


A 

course or course sequence 

J 


Algebra A-i- 



Biology A-i- 

COLLEGE 

English II B- 

U.S. History B-i- 

READINESS 



Work/life experience 
(e.g., military, participation in 
Juried competitions) 


Assessment score on a 
non-cognitive measure 



A - Past learning 
B - Life experiences 
C - Careers 

D - Professional training 



A - Academic motivation 
B - Confidence and self-efficacy 
C - Study skills and work ethic 
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r 


are the implementation considerations that 

the panel believes to be relevant. 

1. Explore potential additional mea- 
sures that faculty and administra- 
tion beiieve might inform piacement 
decisions. Investigate the feasibility of 
each of these, and retain those that can 
be measured reliably and with minimal 
additional cost (possible examples include 
high school CPA, the highest math course 
taken, the number of years since gradu- 
ation, high school graduation status, and 
scores on college entrance examinations). 
Then, develop one or more placement 
rubrics based on the selected measures, 
and use administrative data to investi- 
gate the effect of the different rubrics on 
placement decisions. Use this information 
to propose a specific multiple measures 
policy. 

2. Use a pilot period to study the piace- 
ment rates and success rates of 
students invoiving different piace- 
ment methods at your institution. For 

example, use existing administrative data 
at your institution to develop one or more 
alternative placement rubrics. Then, ran- 
domly assign students to be placed using 
the new multiple measures rubric(s) or 
your college’s current placement method. 
After random assignment, determine if 
student success rates varied as a func- 
tion of placement method. For example, 
you might examine the course pass rate 
for students placed into college algebra 
using your current placement method 
compared to the course pass rate for 
students placed into college algebra using 
an alternative method. Finally, use the 
findings to modify the proposed policy as 
needed. The Research and Planning (RP) 
Group’s Multiple Measures Assessment 
Project offers some resources for set- 
ting up pilot studies, including a “Getting 
Started Guide.’’^^ As needed, partner with 
independent researchers from universities 
and contract research firms for additional 


capacity and expertise on the design and 
conduct of these studies. 

3. After implementation, investigate 
the effects of the multiple measures 
policy on desired student outcomes 
over time. Gontinue to monitor the 
effects of the placement policy, and use 
new information to adapt, change, or 
expand implementation of multiple mea- 
sures policies and practices. 

4. Consider how college-readiness poli- 
cies and practices in your state and 
local area impact the parameters of 
decision-making and implementation 
of multiple measures. For example, as 
states implement new college-readiness 
assessments associated with implementa- 
tion of the Gommon Gore State Standards, 
validation studies of these assessments 
then may offer information to guide the 
potential effectiveness of using them as a 
placement measure. 

Potential roadblocks and 
suggested approaches 

Roadblock 1.1. Using muitiple measures for 
piacement is resource-intensive. 

Suggested Approach 1.1. a. Scale up a 
pilot effort, with support from leadership 
for larger-scale implementation, to gradually 
influence changes in placement practices. Per- 
haps the most significant barrier is the lack of 
resources to pay for a more comprehensive 
assessment process. For example, one faculty 
member at an institution that was incorpo- 
rating multiple measures estimated that, to 
make recommendations, each student file 
required a 10-minute review, totaling about 
40 hours for 250 students. 

One strategy for initiating time- and 
labor-intensive systemic change is to 
begin by convening a work group to lead 
small-scale piloting and plan for gradual 
expansion. For example, the University of 


( 15 ) 


Recommendatiori 1 (continued) 


Wisconsin-Marathon County began using mul- 
tiple measures to place students with certain 
risk factors and then expanded to a larger 
population. Similarly, Long Beach City Col- 
lege (LBCC) first offered alternative measures 
for placement in fall 2012 with 933 students 
from the Long Beach Unified School District, 
expanded to two additional large districts 
the next fall, and two more the year after.^® 
Piloting also enabled gradual adjustments, 
refinements, and adaptations across colleges 
in California and in Wisconsin’s university 
system. 

Moreover, the pilot efforts from LBCC helped 
to guide expansion of the multiple measures 
implementation across colleges in California. 
LBCC conducted and shared its pilot research 
with other colleges as part of the Student 
Transcript-Enhanced Placement Study, which 
found that high school grades were better 
predictors than placement tests (and other 
measures) of students’ passing their first 
attempted courses in English and math.^^ Sub- 
sequently, a consortium of California colleges 
(California Partnership for Achieving Student 
Success) have implemented and studied the 
effects of using multiple measures for stu- 
dents’ academic outcomes.^® 

To maximize successful, long-term implemen- 
tation of new multiple measures practices and 
policies that begin with a pilot effort, leader- 
ship strategies should be inclusive, collabora- 
tive, and adaptive. To translate pilot efforts 
to larger-scale change, leaders must also 
consider how to best leverage and maximize 
scarce resources in each step, while investing 
strategically in implementation of new place- 
ment measures. 

Suggested Approach 1.1. b. Consider 
additional measures already collected by your 
institution or those that are easily assessed. 
Certain measures (such as college entrance 
tests, high school CPA, and the number of 
years since high school graduation) may 
already be collected by your institution and, 
therefore, represent low-cost information that 


can be used to potentially improve placement 
decisions. 

Roadblock 1.2. Lack of support from 
key institutional staff can inhibit robust 
implementation. 

Suggested Approach 1.2. Key institutional 
staff need to be motivated and engaged for 
effective reform to occur.^® Broad sharing 
of information about the severe error rates 
associated with the use of single-assessment 
measures for placement in courses may moti- 
vate and engage faculty and staff to reform 
institutional practices. 

In addition to sharing research about the 
limitations of single-test placement, faculty 
and administrators can also provide evidence 
regarding the advantages of using multiple 
measures to increase probability of more 
accurate placement for a greater number of 
students, without negatively affecting student 
success rates. Duffy et al. (2014) reported 
that a multiple-measures working group in 
North Carolina met with faculty on several 
occasions to explain the research conducted 
by the Community College Research Center 
on this topic as a way of supporting faculty in 
adopting new placement practices and using 
multiple measures. When available, sharing 
the results of institutional research from pilot 
studies can garner support among faculty 
and staff. 

Roadblock 1.3. Data availability and qual- 
ity may be varied, depending on the specific 
assessment measures to be used. 

Suggested Approach 1.3. Access avail- 
able data for the targeted student population 
for the multiple measures to be used and, 
as a secondary strategy, consider the use 
of students’ self-report data. Access to high 
school transcript data by many open-access 
institutions has increased with the continued 
development of P-20 longitudinal, educa- 
tional data systems in states. However, some 
institutions may still have limited access to 
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transcript data, or find the data to be of low 
quality for their student population. In the 
absence of official records or other desired 
data, using students’ self-report data should 
be considered. 

In cases where transcript data is not easily 
shared or accessible, student self-report data 
may offer some insight to the student’s aca- 
demic preparedness, even if that information 
is relatively unreliable. According to Research 
for Action (2014), some colleges in Califor- 
nia use measures, such as student surveys 
and interviews, to gather information about 
highest math course completed and grades 
in last English and math class completed. 
Several testing companies have also devel- 
oped and evaluated assessments to measure 
students’ persistence, dependability, motiva- 
tion, and ability to work with others. ACT, 

Inc., developed a standardized measure of 
student motivation, social engagement, and 
self-regulation, aligned with its college readi- 
ness standards. The assessment. Engage, is 
a 30-minute, multiple choice test designed to 
measure psychosocial attributes and study 
skills associated with college readiness. Little 
or no evidence has been reported to show 
the effectiveness of using these non-cognitive 
measures for students’ course placement. 

While some researchers have identified con- 
cerns with students inflating their responses 
to socially desired qualities, Schmitt and col- 
leagues (2011) found that students’ inflation 
on responses did not “impact the variability 


or validity of responses” (p. 12). Because 
research findings on this issue vary, the 
expert panel recommends cautiously moni- 
toring response quality from self-report data 
in high-stakes situations. 

Roadblock 1.4. Developing new algorithms 
for placement, merging data sources, and 
supporting use of new data for placement 
reguires technical expertise that many open- 
access institutions do not currently have. 

Suggested Approach 1.4. Faculty and 
staff who have strong interest and technical 
aptitude can be supported through profes- 
sional development, encouragement, and 
compensated time for leading and perform- 
ing the work involved. Resources and avail- 
able expertise for implementation of new 
measures for placement will vary by state 
and institution. As mentioned previously, for 
example, the RP Croup in California provides 
resources to help institutions get started 
with multiple measures implementation, and 
other states have developed similar exper- 
tise. Institutions should think creatively and 
strategically about how to develop internal 
expertise while investing in external advice 
on the development of new placement poli- 
cies, integration of data sources, and analysis 
of the results from pilot studies. 
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Recommendation 1 implementation checklist 


Recommendation 1 

Use Multiple Measures to Assess Postsecondary Readiness and Place Students. 

Implementation Checklist^” 


Phase I. Define the Initial Measures and Data Sources. 


□ Convene a working group to lead the multiple measures adoption process. 

□ Determine the placement measures with broad stakeholder input and background 
research. 

□ For all measures, find the data source and data match rate for your population (e.g., 
proportion of targeted student population with high school transcript data). 


Phase II. Report Baseline Statistics, Student Eligibility, and Outcomes. 


□ To show where students would have placed using a multiple measures system com- 
pared to the current placement method, produce a report with retrospective cohorts 
of students to show differences in placements and outcomes. In this baseline report, 
include outcomes such as: 

■ Placement into English and math, by level 

■ Proportion of students who enroll, by placement level 

■ Completion of initial college-level course, by placement level 

■ Completion of course sequence, by initial placement 

□ If not a retrospective analysis, produce an analysis of multiple measures place- 
ment with a theoretical cohort of students. Report on potential impacts of the new 
placement rules on a set of students that have already completed their courses, for 
whom success rates are available. Use the same data matching procedures that you 
would use for existing or previously enrolled students. 

□ Apply the new placement rules to new applicants who have not yet enrolled. Decide 
whether the new placement rules will be applied to a specific cohort of students? 
From specific, feeder high schools? Randomly selected subset of students assigned 
to courses by different types of measures? Include a matched comparison group 
that was not placed by multiple measures and was placed by traditional methods. 

□ Complete the data matching for the new set of measures to the targeted student 
population, and define the outcomes to be tracked for students placed by new 
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1 


measures vs. existing policy. Set up the process to track these students by placement 
method and outcome. 

■ Proportion of incoming students placed at transfer level 

■ Enrollment in discipline course (yes/no) and which level 

■ Success rate of students in first course 

■ Successful completion of transfer-level course in first year, second year, third year 

■ Disaggregations by demographics, equity categories 


Phase III. Use Pilot Data and Stakeholder Input to Modify Implementation and Scale-Up. 


□ Use outcome data on the new placement methods compared with traditional place- 
ment methods (from Phase II) to inform revisions to the new placement rules and 
scale-up of multiple measures implementation. 

□ Engage faculty and staff in decision-making and implementation of multiple mea- 
sures policies and practices. 

□ Track results from each phase of multiple measures implementation and modify 
process as needed 
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Recommendation 2: Require or incentivize regular 
participation in enhanced advising activities. 

Advising, guidance, and counseling services have been developed to help students at all levels 
make choices about academic majors, understand the relationship between school and subsequent 
employment, and address a variety of academic and personal issues. Some colleges have created 
more intensive advising experiences, often called “enhanced advising”^^ or “intrusive advising.”^^ 
Enhanced advising replaces the quick, transactional structure of traditional advising (e.g., a focus on 
class schedules, degree requirements, and financial aid procedures) with a more holistic structure in 
which advisors ask deeper questions and engage more with students to help them succeed.^"" Mentor- 
ing programs that aim to build relationships between students and knowledgeable adults on goal- 
oriented academic planning may also be considered as enhanced advising. 


Researchers have estimated typical advisor-to- 
student ratios for large, community colleges 
and regional open-access institutions to be 
about 600:1.^^ In a study of the Opening Doors 
program in Ohio, implementation of enhanced 
advising meant a reduction of the advisor-to- 
student ratio to 150:1, compared to the typical 
1,000:1 in that college. Scrivener and Weiss 
(2009) also emphasized the importance of 
addressing any underlying structural defi- 
ciencies that limit more personalized, holistic 
advising practices. 

To address costs associated with reducing 
advisor-to-student ratios, postsecondary 
institutions have adopted a range of technol- 
ogy-mediated advising tools and systems as a 
way of efficiently advising large numbers of 
students. Advising technologies may empower 
students to more actively engage with advi- 
sors, and provide tools to support effective 
“advising-as-teaching” approaches, but not 
supplant direct, interactive contact with advi- 
sors.^^ In a study of community college stu- 
dents’ attitudes towards technology-mediated 
advising systems, Kalamkarian and Karp 
(2015) found that students seek an interactive 
approach with advisors, particularly for multi- 
semester course planning and other complex, 
cognitive tasks. 

Summary of evidence: 

Moderate Evidence 

The panel defined the level of evidence for 
this recommendation as moderate. 


This recommendation is supported by three 
studies that met standards without reserva- 
tions (Cousert, 1 999; Scrivener & Weiss, 

2009; Visher, Butcher, & Cerna, 201 0). 

Cousert (1 999) and Visher et al., (201 0) inves- 
tigated the impact of one-semester mentoring 
programs, and Scrivener and Weiss (2009) 
evaluated effects of an incentivized advising 
program offered over a two-semester period. 
The large majority (79%) of the three study 
samples (totaling 3,563 students) consisted 
of students enrolled in developmental educa- 
tion courses within the study period. 

Credit accumulation was assessed in all three 
studies. Meta-analytic results indicated a 
small, statistically significant effect in favor of 
the group of students who received enhanced 
advising on college-level credit accumulation 
(g = -1-0.1 1, p< .001). To contextualize this 
effect, if comparison group students earned 
an average of 6.0 credits during the program 
semester, then the enhanced advising group 
would earn about 6.7 credits on average. 

This means that for approximately every 
1 00 students who receive enhanced advis- 
ing, 23 students will complete one additional 
3-credit-hour course. 

Academic achievement was assessed in two 
of the three studies. Meta-analytic results 
indicated a non-statistically significant effect 
for academic achievement {g = -i-O.Ol , p = 

.73). Progress through developmental educa- 
tion was assessed in two of the three 



Recommendation 2 (continued) 


studies;^® the effect was positive, but not 
statistically significant (^ = +0.07, p= A8). 

Other effects assessed in these studies 
tended to be small and not statistically 
significant. For example, Scrivener and Weiss 
(2009) found a small but non-statistically 
significant positive effect on initial enrollment 
and a small but non-statistically significant 
negative effect on degree attainment (3% 
attainment in the comparison group versus 
2% in the advising group, from the first pro- 
gram semester through third post-program 
semester). Appendix D Table 2 has more 
detail about these studies. 

How to carry out the 
recommendation 

An important aspect of enhanced advising is 
a dedicated and lasting interaction between 
an advisor and a student. The purpose is to 
interact personally with the student to foster 
learning, encourage course completion, and 
decrease institutional barriers that limit or 
prevent student participation in the intellec- 
tual and social life of the college. 

1. Recruit and train advisors. It is crucial 
to encourage and gain the participation of 
a critical mass of advisors. The advisors 
may be college counselors, volunteers 
from college staff, or professors. Strate- 
gies for increasing the number of advisors 
may involve a combination of negotiating 
with existing faculty and staff to take 
responsibility for enhanced advising roles, 
along with hiring of additional profes- 
sional advisors. 

The program should include advisor 
training, and specific enhanced advising 
techniques should be discussed in the 
training. The National Academic Advising 
Association offered essential steps and 
components for developing the content 
of effective advisor training which include 
the college’s concept and definition of 


the advising process, the informational 
aspects of advising, and the relation- 
ship skills involved in supporting stu- 
dents’ decision-making and planning. "" 
Researchers have also found that train- 
ing faculty and staff in how to conduct 
“advising-as-teaching” is a promising 
way to engage with students, and utilize 
technological advising tools as supports 
to facilitate this advising instruction. 

Furthermore, training should establish 
expectations for the frequency and 
intensity of contact with students, and 
provide supports for accomplishing 
this. Visher and colleagues (201 0), as an 
example, suggested providing a variety 
of programming, including a training 
manual to advisors with student services 
information and frequently asked ques- 
tions, a monthly “brown-bag” training 
or refresher sessions, incorporating 
team-building exercises between faculty/ 
staff-student pairs, or a final semester 
gathering to recognize the hard work and 
dedication of the advisors and students. 

2. Require or incentivize in-person 
advising meetings. While students may 
favor using technology-mediated tools 
for simple, administrative tasks, they 
generally find in-person meetings to be 
more effective for support and encourage- 
ment, and for productively engaging in 
goal-setting, complex educational plan- 
ning, and problem-solving. Figure 2.1 . 
offers a sample planning tool used to 
facilitate enhanced advising services at 
Ivy Tech Community College in Cousert 
(1 999). While students were not required 
to participate in mentoring as part of 
the Cousert (1 999) study. Ivy Tech Com- 
munity College requires all new students 
to meet with an academic advisor before 
registering for classes. The Beacon Men- 
toring Program similarly offered support 
for accessing enhanced advising in the 
context of students’ required mathematics 
classes. 
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Incentivizing participation in in-person 
advising, such as by providing a small 
stipend contingent on attendance, is 
another way to bolster attendance. For 
example, Ohio’s Opening Doors program 
offered students $1 50 per semester for 
attending at least two in-person advising 
sessions. It may be helpful to prioritize 
supports for at-risk students, who are 
especially susceptible to discontinuing 
due to difficulties navigating the enroll- 
ment process. 

3. Require or incentivize students to 
meet with advisors frequently and 
over a long(er) period of time. One 

advisory meeting is generally not enough 
time to advise students, particularly 
for multi-semester course planning and 
career guidance. In the Opening Doors 
Program in Ohio, students had an average 
of 8 contacts with their advisors during 
the first and second semesters.''® Offer- 
ing regular meetings, before, during, and 
at the end of semesters is one way to 
encourage frequent meetings. In addition, 
advisors may meet with groups of stu- 
dents, such as during a certain class or at 
a structured time, to facilitate the commu- 
nication of information more efficiently. 

As reported in Visher et al. (201 0), the 


Beacon Mentoring Program encouraged 
advisors to plan short information ses- 
sions to be presented during a develop- 
mental English course, which occurred 
2-3 times per semester. Advisors also met 
with students throughout the semester. 
The Opening Doors Program encouraged 
advisors to meet with students at least 
twice per semester for two semesters in a 
row.'"' 

4. Use early alert systems to iden- 
tify the students who most need 
enhanced advising. Early alert and 
intervention systems are designed to 
identify and support students at risk of 
attrition and improve their retention, 
persistence, and success.''® Early alert 
systems, or early warning systems, iden- 
tify students in need of an intervention, 
which often includes enhanced advising. 
Though no studies on early alert systems 
met WWC standards for inclusion in this 
practice guide, Zachry Rutschow and 
Schneider (201 1 ), suggested that these 
systems can be effective if they bring 
students in contact with needed services 
(though note that mere alerts, without 
subsequent resource mobilization, are 
unlikely to be effective). 
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Figure 2.1. Sample enhanced advising planning tool 


Sample 

Enhanced Advising Tool''® 


ADVISOR’S ROLE AND PREFERRED CONTACT INFORMATION 
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My Goal or Concern My Plan of Action 
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Note: Your institution may already have advising tools available through software already purchased 
or designed that has these components. If so, evaluate how effectively the current system works to 
facilitate ongoing communication between advisors and students. If not, consider developing a 
shared framework, such as this, to guide advising sessions. 
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Potential roadblocks and 
suggested approaches 

Roadblock 2.1. Decreasing the student- 
to-advisor ratio means finding and training 
more advisors. 

Suggested Approach 2.1. Recruit advi- 
sors from various sources in addition to 
hiring trained specialists. For example, the 
Beacon Mentoring program recruited advi- 
sors not only from the counseling and advis- 
ing department, but also from student life, 
research/technology, special academic/career 
programs, assessment and placement, admis- 
sions and records, outreach and orientation, 
administration, and financial aid. Faculty 
provided advisors with in-class time to make 
at least three presentations per semester, 
and also gave advisors data on students’ 
attendance and grades so that advisors could 
intervene with struggling students. 

The student-to-advisor ratio is high at most 
community colleges, and many students may 
then make decisions that are poorly suited 
to their needs, academic goals, and career 
objectives. In relation to a typical student-to- 
advisor ratio of about 600:1 in community 
colleges, the National Academic Advising 
Association recommends decreasing the advi- 
sor load as low as possible (without recom- 
mending a specific ratio; see Robbins, 201 3). 
In the Opening Doors Demonstration in 
Ohio,^’ the student-to-advisor ratio was 81:1 
at Owens Community College and 1 57:1 at 
Lorain Community College. In this program, 
the reductions in student-to-advisor ratio 
were achieved by hiring additional counseling 
staff in the student services divisions of the 
colleges. 

To decrease the advisor load, trained college 
counselors and advisors should not be the 
only ones to provide assistance to students. 
Well-trained college faculty and staff should 
also provide advising and mentoring to 


at-risk students (for an example of how 
faculty and staff were used in this way, see 
Visher et al., 201 0). While faculty and staff 
who volunteered to serve as mentors in the 
Beacon Mentoring program did so as part of 
their Job, engaging additional faculty in staff 
in new advising roles may require additional 
incentives or contract requirements. 
Importantly, decreasing institutional barriers 
can be accomplished by simply discussing 
challenges in addressing the procedural 
aspects of college. For example, staff or 
faculty with experience determining college 
courses can be helpful to students who have 
difficulty making these decisions. 

Roadblock 2.2. It may be difficult to get 
students to attend advising sessions. 

Suggested Approach 2.2. Due to schedul- 
ing issues, community college students may 
find it difficult (and some may be unwilling) 
to participate in one-on-one advising ses- 
sions. In addition, the advising sessions may 
be difficult for students to set up or keep 
track of. Be attentive to potential scheduling 
issues when establishing the schedule for 
enhanced advising sessions. Scheduling of 
enhanced advising sessions should consider 
students’ work and course schedules. Also, 
some enhanced advising activities may be 
appropriate for class time (especially on the 
first day of classes; see, for example, Visher 
et al., 201 0), and some may appropriately 
take advantage of video conferencing ser- 
vices, such as Skype and FaceTime. Finally, 
consider setting up an electronic reminder 
system (e.g., via email or text message). 

Roadblock 2.3. Advisors may not always 
be fully eguipped to deal with the range of 
issues presented by students. The range of 
issues faced by at-risk students is large and 
may require a skill set similar to that needed 
by counselors. College advisors may not be 
well-equipped to handle such problems and 
may inadvertently allow students to return to 
recurring behaviors. 
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Suggested Approach 2.3. Train and pro- 
vide guidance to advisors regarding the range 
of services available on campus. Scrivener 
and Weiss (2009) provided a specific training 
module along with certification, while Visher 
et al. (201 0) suggested providing a training 
manual to advisors with student services 
information and frequently asked questions, 
offering monthly “brown-bag” training or 
refresher sessions, and utilizing team-build- 
ing exercises between faculty/staff-student 
pairs. The nature of this training should 
ensure that advisors understand the range of 
services available on campus. 

Roadblock 2.4. Coordination and communi- 
cation across student services is ineffective or 
underdeveioped. Advisors, professors, tutors, 
and staff sometimes fail to communicate or 
follow-up to all invested parties regarding 
students’ needs, in relation to their academic 
and career goals. 

Suggested Approach 2.4. Train advisors 
to work with students and other resource 
providers to connect students with multiple 
solutions and supports as needed so that 
they may be successful with short- and long- 
term goals agreed upon in an action plan. 
Cousert (1 999) outlined the approach used 
by Ivy Tech State College in central Indiana, 
whereby faculty mentors work with students 


on an individualized action plan. The men- 
tors document referrals and resources for 
the students to access in the plan and then 
follow-up with the students to determine 
their results in accessing those supports. The 
action plan documents students’ concerns, 
actions to be taken, and timeline for complet- 
ing the actions. At South Texas College, the 
Beacon Mentoring Program provided students 
with knowledge and support of campus 
resources, such as tutoring, advising and 
counseling services, financial aid, and priority 
registration. To provide support before they 
failed or dropped out of the class, mentors 
communicated with math instructors about 
struggling students and also instilled a cli- 
mate where they were the “go-to” person on 
campus for students’ questions.” Depending 
on the type of advising structures, communi- 
cation mechanisms, and resources available, 
campuses can be resourceful and strategic 
about how to personalize and maximize 
students’ knowledge and access of needed 
campus resources. 


(25 ) 


Recommendatiori 2 (continued) 


] 


Recommendation 2 implementation checklist 


Recommendation 2 

Require or Incentivize Regular Participation in Enhanced Advising Activities. 

Implementation Checklist^° 


Phase I. Planning 


□ Set goals, objectives, and measures of success for the enhanced advising process. 
For example, to which degree are your institution’s advising goals focused on: 

■ Course scheduling/registration 

■ Compliance advising 

■ Academic planning 

■ Developmental advising and mentoring 

□ Engage existing and potential advisors in determining advising goals and measures 
of success. 

□ Determine tangible administrative supports and commitments needed to achieve 
enhanced advising goals. For example, in which ways do you need: 

■ Technology-based advising system for administrative advising tasks 

■ Additional full-time-equivalent staff 

■ Financial incentives for faculty to expand advising time 

■ Financial incentives for students to regularly participate in advising sessions 

■ Resources for training advisors to achieve the new goals and outcomes 

■ System for providing and monitoring support to students identified as at-risk 
through early alert processes 

■ Time allotted in courses for advising functions 

■ Changes to faculty evaluation process to include changes to advising 
expectations 

■ Resources and time to recognize the work and accomplishments of advisors and 
students 


Phase II. Implementation 


□ Use your institution’s most experienced advisors to recruit, hire, and engage more 
advisors. 
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□ Create and use a comprehensive advising manual in training new advisors and in 
refresher sessions. 

□ Provide training and support, so that new advisors can address many college- 
related issues with their students. 

□ Determine how the new advising methods will be implemented, based on the 
agreed-upon advising goals, measures of success, and process agreed upon by 
faculty, staff, and administration. 

■ Track participation and results from in-person advising meetings. 

■ Support the use of technology-mediated advising tools for simple, administrative 
advising tasks. 

■ Provide incentives for students to regularly attend in-person advising sessions. 

■ Expect and plan for regular, frequent meetings with advisors. 

■ Offer small group advising sessions to support peer connections on shared 
concerns. 

■ Identify and support at-risk students early. 

D Consider piloting the new advising method with a smaller population before scaling 
up the process for all students. 


Phase III. Evaluation and Expansion 


□ Evaluate student outcomes achieved from enhanced advising methods compared to 
traditional advising methods, such as credit accumulation, retention, and academic 
achievement. 

□ Using the evaluation criteria your institution established for advisors, conduct evalu- 
ations of advisors that will guide conferral of incentives and rewards. 

□ Provide recognition to the efforts of students and advisors through celebrations and 
tokens of appreciation. 

□ Evaluate the effectiveness of various components of the advising system such as 
student incentives, technology-mediated advising tools, advisor training, and admin- 
istrative supports. 

□ Share report findings on the degree to which the enhanced advising program 
achieved its goals and desired outcomes with key stakeholders. 

□ Use evaluative data to modify the enhanced advising processes, allocate additional 
resources, or expand enhanced advising to additional student populations. 
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Recommendation 3: Offer students performance- 
based monetary incentives. 

Performance-based incentives are monetary awards that are disbursed to students based on meet- 
ing specific academic benchmarks. These awards supplement students’ financial aid packages, which 
may he based on need (e.g., Pell grants) or past achievement (e.g., state merit aid grants). In certain 
instances, the performance-based awards may reduce the number of loan dollars taken and grant 
dollars awarded, hut they usually result in a net financial gain for students (see Patel & Valenzuela, 
2013). The short-term goal of such initiatives is to encourage students to perform better in (and suc- 
cessfully complete) their classes. A longer-term goal is to support students’ progress through devel- 
opmental and degree requirements to increase degree attainment.^"^ 


Incentive programs can vary in several 
ways, including the amount of the incentive 
and the number of semesters for which 
students are eligible for it. In the studies 
summarized below, the incentive amounts 
ranged from about $600 to $1 ,500 per 
semester, and students were typically 
eligible for 2 or 3 semesters. Generally, 
students are allowed to use the awards for 
any purpose (that is, not necessarily for 
educational expenses; Scrivener & Coghlan, 
2011). Incentives are usually distributed 
at predetermined time points throughout 
the semester, rather than in one lump- 
sum payment. The rationale behind this 
disbursement structure is that it encourages 
students to meet program goals (e.g., 
maintaining a minimum level of enrollment). 

Summary of evidence: 

Moderate Evidence 

The panel judged the level of evidence sup- 
porting this recommendation to be moderate. 

Six randomized controlled trials, which 
involved nearly 8,000 participants, examined 
the effects of performance-based monetary 
incentives. About two-thirds (67%) of the stu- 
dents across these six study samples enrolled 
in developmental education courses within 
the study period. Three of these studies 
examined the effects of performance-based 
monetary incentives alone or with minimal 
additional services” and three studies exam- 
ined performance-based incentives with 


additional services (i.e.. Miller et al., 2011, 
coupled incentives with enhanced advising; 
Patel & Valenzuela, 2013, coupled incentives 
with enhanced advising, tutoring, workshops, 
and other supports; Sommo et al., 2014, 
combined incentives with supplemental math 
lab instruction). 

Offering incentives oniy. Three random- 
ized controlled trials were identified that 
investigated the impact of performance-based 
incentives alone or offered monitoring advis- 
ing support in addition. All three studies 
met WWC group design standards without 
reservations. In two of these studies,” most 
students (85 percent, across the two study 
samples) were in developmental education. 

In the other study,” about 40% of students 
in the study sample took at least one devel- 
opmental education course. (See Appendix D 
Table 3.1 for further details.) 

Across these three studies, incentives were 
associated with a small but statistically signifi- 
cant impact on the first instance of enrollment 
after randomization to condition {g = - 1 - 0 . 08 , 
p < .01). To contextualize this effect, if the 
base rate of enrollment in the first academic 
term was 94% in the comparison group, about 
94.6% of students in the intervention group 
would have initially enrolled. Two studies” 
assessed the effect of performance-based 
incentives on academic achievement; this 
meta-analytic effect was positive, but not 
statistically significant {g = -1-0.09, p = .09). 
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All three studies evaluated the effects of 
providing performance-based incentives on 
college-level credits earned, and the meta- 
analytic results indicated a positive and 
statistically significant increase {g = -i-O.lO, p < 
.001) in credits earned (after 7 or 8 semesters, 
depending on the study). To contextualize 
this effect, if the typical comparison student 
earned 14 college-level credits during the first 
year, the typical intervention student would 
have earned about 15.2 credits. This suggests 
that for every three students in the interven- 
tion condition, one additional student would 
complete an additional 3-credit-hour course 
over 7-8 semesters. 

Finally, two studies assessed the effect of 
performance-based incentives on degree 
attainment.^® The meta-analytic results indi- 
cated a non-statistically significant positive 
increase {g = -1-0.06, p = .07). 

Combining incentives with other sup- 
ports. Combining incentives with other 
supports. Similar to the meta-analytic effects 
observed in incentives-only studies, providing 
performance-based incentives with additional 
supports also had a small but statistically 
significant impact on enrollment {g = - 1 - 0 . 08 , 
p < .03) across the three studies.®® Contextu- 
alizing this effect leads to the same conclu- 
sion as the enrollment analysis above in the 
incentives-only grouping: if the base rate of 
enrollment was 94% in the comparison group, 
about 94.7% of students in the intervention 
group would have enrolled. 

Sommo et al. (2014) was the only one of these 
studies to examine student progress through 
developmental education (i.e., later earning 
college-level credit in their area of remedia- 
tion). In this study, developmental math stu- 
dents in the intervention condition were more 
likely than developmental math students in 
the comparison group to pass college-level 
math (32% versus 26%, p < .05). 


Academic achievement was measured by two 
studies in this group.®' In both cases, aca- 
demic achievement was measured by assess- 
ing the proportion of students who earned a 
C average or better across their coursework. 
Meta-analytic results indicated a positive and 
statistically significant intervention effect {g = 
-1-0.12, p< .01). To help understand this effect, 
if 50% of students in the comparison group 
earned at least a C average or better, on aver- 
age, 54% of students in the intervention group 
would earn at least a C average or above. 

All three studies assessed the effect of 
performance-based incentives with additional 
supports on credit accumulation, but the 
meta-analytic effect was not significant {g = 
+0.03, p=.30). 

Finally, two studies assessed the effect of per- 
formance-based incentives on degree attain- 
ment. The meta-analytic results indicated a 
positive, statistically significant impact {g = 
+0.08, p < .02). To contextualize this effect, 
if 33% of students in the comparison group 
graduate, 36.1% of students in the interven- 
tion group would graduate. (See Appendix D 
Table 3.2 for further details.) 

How to carry out the 
recommendation 

The primary purpose of this intervention 
is to support students by tying monetary 
incentives, which do not replace financial aid 
but are given in addition to it, to achieving 
academic success. See Figure 3.1. for some 
tips on how to budget for performance-based 
monetary incentives. 

1. Design and structure payments of 
students’ performance-based incen- 
tives to yieid students a net hnanciai 
gain. Since many financial aid packages 
are tied to a student’s level of unmet need, 
an important consideration is the impact 
that this monetary incentive will have on 
the overall level of aid. Mayer, Patel, and 
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Gutierrez (2015a) suggested that financial 
aid offices work with students and admin- 
istrators to ensure that any incentives 
provide a net financial gain to participat- 
ing students. 

2. Distribute payments incrementaiiy, 
according to specific desired perfor- 
mance benchmarks. To encourage and 
motivate students to reach academic mile- 
stones, distribute payments throughout 
the semester or year. Midterms and final 
exams are good candidates for disburse- 
ment periods. 

3. Target incentives to iow-income 
students. The evidence suggests that 
providing monetary incentives has posi- 
tive effects for low-income students and 
various at-risk student populations (such 
as Latino male students with delayed 
entry to college). Programs should con- 
sider prioritizing these students. 

4. Provide student support services, in 
conjunction with performance-based 
monetary incentives. Many students in 
developmental education may find it dif- 
ficult to make time for additional campus 


supports without a specific incentive — 
given their work, family, and academic 
demands. Monetary awards can help 
incentivize students to take advantage of 
support services (e.g., academic advis- 
ing, tutoring, mentoring) that will help 
them succeed. For example, the Adelante 
Performance Award Program linked pay- 
ments to utilizing support services and 
academic milestones, thereby achieving 
high student attendance for supports, 
such as program orientations, advising 
sessions, tutoring services, and work- 
shops. ®^These supports are academically 
useful for students, and the personal 
contact made during these interactions 
may also help students feel motivated, 
encouraged, and accomplished when they 
succeed. 

As a useful resource, MDRC published a 
technical assistance guide for developing and 
implementing performance-based scholar- 
ships which highlights key strategies for 
assembling key stakeholders, funding your 
program, creating a budget and partnerships, 
as well as aspects of important elements of 
designing, implementing, and monitoring the 
program.®^ 
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Figure 3.1. How to budget for performance-based monetary incentives 


How to Budget for Performance-Based Monetary Incentives^'* 

Imagine your institution wants to encourage high-risk students to earn 12 or more 
credits per term. You have decided to offer 100 students a performance-based 
monetary incentive in which they can earn up to $1,000 over the term. How much do 
you need to budget to achieve this goal? 

You plan to pay the first installment of $100 to students who enroll in 12 or more 
credits. Your institution expects that 90 percent of students will earn the first payment. 
The cost will be 90 students x $100 = $9,000. 

You plan to pay the second installment of $900 to those who complete at least 12 
credits with a 2.0 GPA. Your institution expects that 60 percent of those 100 students 
will earn the second payment. The cost will be 60 students x $900 = $54,000. 

The budget for the students' performance-based monetary incentives is thus $63,000. 
Because your institution will end up paying for incentives for fewer students than the 
number of students eligible for the incentive, your institution may be able to incentivize 
more students. 

Projections of how many students will earn each incentive payment should guide 
budgeting. Administrative costs to manage the program should also be figured into the 
budget. 


Potential roadblocks and suggested 
approaches 

Roadblock 3.1. Performance-based incentives 
are expensive to offer. 

Suggested Approach 3.1. Consider costs 
in relation to potential enrollment increases 
and successes from the program. Although 
the initial financial commitment to these 
programs may result in difficult allocation 
decisions, institutions may benefit from 
increases in student enrollment over time if 
enough students are successful. Furthermore, 
Patel and Rudd’s (2012) analysis suggests 
that the cost to implement such programs 
might decrease as a function of the number 


of students enrolled (i.e., administrative costs 
per student are lower when more students 
are enrolled in the incentive program). Fur- 
ther, Mayer and colleagues (2015b) conducted 
cost-effectiveness analyses of six perfor- 
mance-based monetary incentives programs. 
In 2014 dollars, the average direct cost of a 
performance-based scholarship program was 
$2,345 per student for the program’s dura- 
tion. Mayer and colleagues (2015b) reported 
that about 72% of the direct cost was direct 
payments to students, about one-fourth (26%) 
was program administration costs, and 2.5% 
of the cost was additional student services. 
They found that while the performance-based 
monetary incentives required substantial 
financial investment, the average cost per 
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degree was about the same for students 
who received the intervention compared to 
students who did not have the opportunity to 
earn performance-based monetary incentives. 

In addition, the costs of the performance- 
based monetary incentives will likely be a 
small proportion of students’ total financial 
benefits package. For example, in Ohio, 
students received an average of $765 in 
performance-based incentives, compared 
with $7,947 in total financial aid in the same 
year.®^ Even though proportionately the 
performance-based monetary incentives are 
a small investment, the opportunity to earn 
performance-based monetary incentives 
appears to result in small, positive effects — 
both short- and long-term. 

Roadblock 3.2. Institutions lack staff mem- 
bers who can be dedicated to running these 
programs. 

Suggested Approach 3.2. These programs 
entail new processes that require dedicated 
staff to oversee them to be effective. If possi- 
ble, create a new position to oversee the pro- 
gram or utilize “student success coordinators” 
where available. The duties of this person or 
persons should include monitoring transcripts 
to determine whether students had met the 
award criteria and distributing payments to 
students. This position could be housed in 
the financial aid office or the student services 
office.®® If resources are not available to hire 
dedicated staff, other individuals within the 
school could also serve as monitors (e.g., 
counselors) and payment conduits (financial 
aid officers or departmental staff). 

Roadblock 3.3. Additional monetary awards 
may decrease the financial aid packages that 
students might otherwise receive. 


Suggested Approach 3.3. Because many 
low-income students have unmet financial 
need after financial aid and expected family 
contributions are calculated, performance- 
based incentives can be a source to fill this 
gap rather than extra work hours, taking 
fewer courses, or taking out more loans. 

After financial aid is calculated, students 
without remaining unmet need may not be 
able to receive money from incentives due to 
federal, state, and institutional policies that 
restrict funding beyond students’ level of 
unmet need. One way to address this issue 
is to target students with larger amounts of 
unmet financial need. Another strategy that 
institutions can use to ensure a net financial 
gain for students is to reduce loans when a 
student earns a performance-based monetary 
incentive. In the case of students who do not 
earn the anticipated money from incentives to 
fulfill their unmet need, financial aid directors 
should make sure that students have access 
to loans to fill the gap. Financial aid offices 
should develop and apply procedures for not 
reducing students’ grant aid in the event of 
aid overages.®^ 

Because students’ financial aid awards could 
be reduced if they receive additional fund- 
ing through performance-based incentives, 
administrators and financial aid officers 
should work together to ensure students 
experience a net increase in aid. For example, 
the nonprofit education research organization 
MDRC worked with Hillsborough Community 
College (Tampa, FL) financial aid staff to 
ensure that the Mathematics Access Perfor- 
mance Scholarship recipients did not experi- 
ence a negative impact on their financial aid 
packages, using strategies described above.®® 
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Recommendation 3 implementation checklist 


Recommendation 3 

Offer Students Performance-based Monetary Incentives. 

Implementation Checklist^^ 


Phase I. Planning and Design 


□ Assemble key stakeholders for program funding, design, and implementation and 
identify an administrative program champion (such as the vice president of aca- 
demic affairs or dean of student services). 

□ Identify sources of funding for the performance-based monetary incentives among 
college and community foundations, college departments, and state programs. 

□ Create a budget for program support (i.e., staff time, supplies, materials) and stu- 
dent incentive budget projections (based on targeted population, anticipated level 
of participation, and incentive amounts). 

□ Once program funding sources and program budgets have been outlined, form 
partnership agreements between funders and the institution to guide program 
financing, implementation, and evaluation. 


Phase II. Implementation and Evaluation 


□ In tandem with funding negotiations, confirm the targeted population of students, 
based on information about which students would most benefit and be prioritized 
for incentives (e.g., students in developmental education courses. Pell-eligible 
students or students with zero expected family contribution, part-time students, 
“non-traditional” students). 

□ In the decision-making, ensure that target population and incentive amounts and 
types will operate as incentives for changes in behavior among students most at- 
risk for not achieving the desired goal, not rewards for students who would have 
already achieved the desired goals without the incentive. 

□ In conjunction with the budget decisions, confirm the types of incentives (cash, gas 
or metro cards, course credit). 

□ Decide upon when and how the incentives will be distributed: 

■ By whom? 

■ How much at the beginning, mid-term, and end of the semester? 
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■ For what student actions and achievements? 

□ Determine the length of the incentive program, based upon information for key 
drop-off points for the targeted population on the desired outcomes (e.g., persis- 
tence, achievement in gatekeeper courses, etc.)- 

□ Identify whether the distribution of incentives will occur during fall and spring terms 
only, or whether winter and summer terms will be added (if on a quarter system or 
intersessions are available). 

□ Based on the program’s theory of change and population, confirm the details of the 
staffing plan, and train and support staff as necessary for effective implementation. 

□ Determine how student outcomes and incentive distribution will be tracked and how 
success of the program will be evaluated. 

□ Celebrate the successes of the program at key intervals, and learn from progress as 
the program is implemented and evaluated. 
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Recommendation 4: Compress or mainstream 
developmental education with course redesign. 

Many colleges have adopted an accelerated developmental coursework sequence In which 
students can more quickly complete developmental education courses and enroll In college- 
level courses7° These reforms are referred to Interchangeably In the literature as “Intensive,” 
“compressed,” “condensed,” or “time-shortened” models/' And, while the specific steps 
employed In the models may vary,^^ the Intent Is always to reduce the length of time students 
spend In developmental education with the aim of Improved student success rates. 


Participation In accelerated developmental 
experiences can minimize the negative 
effects of being placed Into developmental 
education. Edgecombe (2011) noted that If 
students can register for more than one 
sequential course in a semester, they are more 
likely to enroll In the second course, thereby 
improving retention. Accelerated courses that 
mainstream developmental education students 
into college-level work with contextuallzatlon 
or supplemental instruction also help students 
achieve the goals and outcomes of the college- 
level course assignments. Further, Jaggars 
et al. (2015) noted that acceleration may 
promote persistence and academic success 
because the reduced time in developmental 
education also reduces the opportunity for 
external factors, such as work or family 
responsibilities, to hinder students’ success. 

The following three types of accelerated 
instructional models have promising 
evidentiary support in the research literature. 

Compressed courses with curricular 
redesign. This approach involves offering 
developmental courses in a compressed 
schedule so students may either complete 
more levels of coursework in a single term or a 
term of coursework in less than the traditional 
16-week semester. These courses and 
models also adapt curricular, instructional, 
and assessment practices as needed while 
supporting students’ mastery of skills and 
content. Faculty adjust the pacing, workload, 
and instructional strategies to cover similar 
curricular content in a shorter timeframe. 
Examples include the City University of 
New York’s (CUNY) accelerated courses,^^ 


Chabot College’s (California) one-semester 
English redesign. Community College of 
Denver’s FastStart program in math, and 
Broward (Florida) College’s math redesign.^^ 

Mainstreaming with suppiementai 
support. Mainstreaming involves placing 
developmental students into college-level 
courses alongside their non-developmental 
education peers. In doing so, students 
underprepared for college-level work 
may succeed in college-level classes with 
the addition of supplemental support 
services. Supplemental supports may 
include companion classes (whereby the 
course instructor also provides additional 
instruction to students who are identified 
as needing developmental education), lab 
sessions, integrated tutorial support, and 
additional class sessions.^'^ One example is 
the Accelerated Learning Program (ALP) at 
the Community College of Baltimore County.^^ 
Another example is the New Mathways 
Project provides a Tyear statistical reasoning 
pathway for students in social sciences or a 
I -year quantitative reasoning pathway for 
students in science, technology, engineering, 
and math fields paired with a college-level 
student success course. This helps students 
develop strategies for knowledge development, 
self-regulation, motivation, and learning.^^ 

Mainstreaming with contextuaiization 
to students’ academic goais and 
interests. This model simultaneously 
provides students with college-level content 
instruction and integration of developmental 
education skill-building, tailored to students’ 
career, technical, or academic interests. For 
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example, the Integrated Basic Education 
and Skills Training program (l-BEST) geared 
toward basic skills students interested in 
career and technical education majors, 
linked developmental education curricula 
with college-level curricula in a co-teaching 
model, focused on students’ occupational 
or disciplinary interests, and engaged local 
employers as partners.^° Statway, is geared 
toward students interested in humanities 
and social science majors who place into 
developmental math courses. The program 
involves mainstreaming these students into a 
1-year college-level statistics course to provide 
them with foundational skills in statistics, 
data analysis, and causal reasoning.^' 

Summary of evidence: 

Minimal Evidence 

The panel assigned a rating of minimal 
evidence to this recommendation. 

One study that evaluated the effects of a 
compressed, redesigned developmental 
writing course®^ met WWC standards with 
reservations. The researchers found that 
students in an accelerated developmental 
writing course sequence were more likely 
than students in the traditional, longer 
sequence to pass college-level English, to 
have earned more college-level credits after 
3 years, and to have earned a degree after 5 
years. All of these effects were statistically 
significant. This study reported the effects 
of compression of developmental course 
material from 8-12 credit hours to 6-7 credit 
hours at CUNY and provides promising 
evidence to support accelerated writing 
instruction.®^ The following three additional 
studies did not meet WWC group design 
standards, but the panel believes that they 
provide credible evidence to support this 
recommendation. See Appendix D Table 4 
for further details about all four studies. 

Mainstreaming with supplemental 
support: Additional evidence from 
the ALP. Cho et al. (2012) examined the 


effectiveness of the ALP at the Community 
College of Baltimore County. This study 
did not meet WWC group design standards 
because the WWC could not assess baseline 
equivalence between intervention and 
comparison groups. However, the researchers 
used a propensity-score matched sample 
and controlled for a number of observed 
student characteristics in the analyses to 
account for influences of factors, such as 
gender, age, race, and measures of student 
academic background (college placement 
scores in reading, English, and math, median 
household income, several financial aid 
indicators, and employed campus and 
cohort fixed effects). Based on an analysis 
of several cohorts of students followed 
for at least 1 academic year, the authors 
concluded that ALP students outperformed 
non-ALP students on college-level English 
course completion rates and persistence to 
the next year (Table 2, p. 7). While about 
75% of ALP students completed first college- 
level English by the next year, only 39% of 
non-ALP students completed the first college- 
level English course in that timeframe. The 
rate of persistence was also higher for ALP 
students: 64% of ALP participants compared 
to 48% of non-ALP participants persisted to 
the next year. The effects for both of these 
outcomes were statistically significant. 

Mainstreaming with contextuaiization: 
Additionai evidence from Washington’s 
l-BEST program. Zeidenberg, Cho, and 
Jenkins (2010) assessed the effects of the 
l-BEST intervention on students enrolled in 
the 2006/07 and 2007/08 academic years. 

In the l-BEST program (a mainstreaming and 
contextualized instruction intervention), 
college-level occupational courses 
with integrated basic skills content are 
co-taught by a basic skills instructor and an 
occupational instructor. The occupational 
coursework was available in a variety of 
fields, such as health care, office work, 
and skilled trades. This quasi-experimental 
study did not meet WWC standards because 
the WWC could not assess the baseline 
equivalence of the intervention and 
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comparison groups. In the propensity-score 
matched sample, all students in both groups 
were in adult basic education and workforce 
programs (i.e., all students took at least 
one career or technical education course). 
Adult basic education serves students at the 
lowest levels of developmental education 
coursework. Even though the study did not 
meet WWC evidence standards, regression 
analyses controlled for a number of factors, 
including students’ basic skills test scores, 
English proficiency, socioeconomic status, 
race, age, and gender. The researchers found 
substantial differences between college 
credits earned by l-BEST students and the 
non-l-BEST comparison group (Table 6, 
p. 14). l-BEST students earned a mean of 
about 25 college credits; students in the 
comparison group earned only 17. l-BEST 
students also had a substantially higher 
rate of receiving a degree or certificate: 51% 
of l-BEST participants earned a degree or 
certificate by Spring 2009 while just 14% of 
students in the comparison group earned a 
degree or certificate. The effects for both of 
these outcomes were statistically significant. 

Compressed courses with redesign: 
Additionai evidence from Chabot 
Coiiege’s integrated reading and writing 
course. Edgecombe et al. (2014) investigated 
the effectiveness of a 1-semester accelerated 
developmental education course in English 
compared with an equivalent two-course 
sequence. The accelerated course model 
also included a redesigned curriculum and 
pedagogical changes that attempted to 
strengthen the integration of developmental 
reading and writing instruction. This study 
did not meet WWC standards because the 
baseline equivalence of the intervention 
and comparison groups could not be 
assessed. However, regression analyses 
controlled for a number of factors, including 
students’ placement scores, socioeconomic 
background, gender, age, marital status, racial 
and ethnic background, educational goals, 
graduation from high school, enrollment 
status, and prior participation in a learning 
community. Based on the 5-year outcomes. 


the researchers found that 60% of accelerated 
developmental English course-takers passed 
the first college-level English course (i.e., 
completed and received a grade of C or 
higher) compared with Just 40% of students 
who took the longer English developmental 
course sequence. Furthermore, students who 
participated in the 1 -semester developmental 
English course earned an average of 36 
college-level credits while students who 
took the traditional developmental English 
course sequence only earned an average 
of 29 college-level credits. Finally, while 
only 18% of non-accelerated developmental 
English course-takers graduated from 
college, 25% of accelerated English course- 
takers graduated from college. All of these 
effects were statistically significant. 

How to carry out the 
recommendation 

Many individuals working together 
contribute to successful outcomes achieved 
by underprepared college students 
through mainstreaming or compression 
of developmental education courses. 
Administrators, professional developers, 
faculty,®^ articulation officers, advisors, 
and students each have important roles 
to play in ensuring efficient transitions 
for developmental education students to 
succeed in college-level courses. See Figure 
4.1. for an illustration of this concept. 

1. Obtain faculty and leadership 
support for rethinking course 
structures and policies to efficiently 
support students in mastering 
developmental education skills 
and earning college credits. Support 
from faculty and leadership is vital to 
developing and implementing an effective 
accelerated course model, especially if 
the large majority of students are to be 
affected by the changes. Particularly for 
mainstreaming models, colleges must 
establish collaborative structures and 
processes for instructors to communicate 
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with one another and ensure that 
students are mutually supported as 
they complete college-level coursework. 
To begin, assess the existence and 
effectiveness of current strategies for 


accelerating students’ progress through 
developmental education (See Figure 4.2.). 
During planning, anticipate challenges 
and strategize collaboratively to achieve 
optimal success (See Figure 4.3.). 


Figure 4.1. Student success through accelerated coursework 
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Figure 4.2. Inventory of strategies currently in place for accelerating 
students’ progress through developmental education 

Figure 4.3. Planning issues and considerations in adopting compressed 
and mainstreaming developmental education models 


Figure 4.2. 


Inventory of strategies currently in 
place for accelerating 
students’ progress through 
developmental education.^^ 


Do you already have 

□ Self-placement options for students 
to reduce need for placement into 
developmental education? 

□ Compressed developmental educa- 
tion curriculum, with rigorous course 
content aligned with transfer- 1 eve I and 
career/technical fields? 

□ Provision of additional student sup- 
ports integrated with coursework? 

□ Integration of skill-building in Eng- 
lish and math within credit-bearing 
courses, aligned with students’ aca- 
demic and career interests? 

□ Monitoring of student progress at 
regular, frequent intervals based on 
student mastery of content rather 
than seat time? 


Figure 4.3. 


Planning issues and considerations 
in adopting compressed and 
mainstreaming developmental 
education models.^^ 


□ Anticipate key challenges in redesign- 
ing developmental education courses. 

□ Strategize about how to engage each 
stakeholder group in redesigning and 
accelerating developmental education. 

□ Think about who may resist redesign 
of developmental education, if not 
engaged in the process. 

□ Consider the goals and methods for 
engaging stakeholders in redesigning 
developmental education, to maxi- 
mize success. 

□ Know what the best-case scenario 
outcomes are for the redesign and 
key milestones to be accomplished to 
achieve those outcomes. 

□ Think through the worst-case scenario 
outcomes when engaging with stake- 
holder groups and what strategies 
can be used to mitigate the chances 
of these outcomes. 
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2. Provide systematic professionai 
deveiopment and support for facuity 
to impiement new acceierated course 
modeis. New models of acceleration 
generally require systematic professional 
development for faculty and, depending 
on the type of acceleration, faculty 
needs for professional development will 
vary. Principles of faculty professional 
development should address: 

■ How to maintain rigorous college- 
level content and skill development 
while meeting needs of students who 
are not college-ready in all areas. 

■ Ways to best use the institution’s 
resources and the expertise of various 
instructors to compress, mainstream, 
or modularize course content. 

■ Models for integrating accelerated 
course models in mathematics, 
reading, and English, or 
contextualizing courses to students’ 
academic and career interests. 

■ If computer-based tools and programs 
are being adopted, training and 
support for effectively using these 
tools to support accelerated course 
models should be provided. 

3. Offer support to faculty for 
differentiating instruction to 
students of varied ieveis of academic 
preparedness in their ciasses. In 

mainstreaming models, faculty who teach 
introductory college-level courses may 
not be accustomed to differentiating 
instruction for students without college- 
level proficiency. Or, they may not have 
experience tightly coordinating their 
curriculum and instructional delivery 
with other instructors (e.g., those who 
provide supplemental instructional 
support). Developmental education 
instructors need to become familiar with 
the rigor of the college course content and 
assessments and determine how to best 
support students’ success in mastering 


the college-level course material. 

4. Mobilize and communicate targeted 
supports for students’ specific 
instructional needs. When accelerating 
students’ progress in developmental 
coursework, students who are not yet 
college-ready may require targeted 
supports.®® For example, students may 
need help sustaining motivation and 
integrating the skills and knowledge 
learned from module to module when 
using a modularization approach. 

Specific suggestions include: 

■ Decide on the best structure, 
staffing, and allocation of credit for 
supplemental instruction, modularized 
content, and redesigned courses to 
offer necessary targeted supports 

to students of varying skill levels. 

■ Use communication supports — such as 
text-based reminders about upcoming 
deadlines and encouragement to 
keep working — to keep students 
motivated, on track, and primed to 
use existing campus resources. 

Potential roadblocks and 
suggested approaches 

Roadblock 4.1. Students may not 
take advantage of accelerated course 
options and their benefits. 

Suggested Approach 4.1. Faculty should 
work with academic advisors or mentors to 
ensure the appropriate students register for 
accelerated courses. Underprepared students 
may be hesitant to sign up for an accelerated 
course in a topic area in which they consider 
themselves weak or have been told they 
need remediation.®® Faculty and support 
staff should develop and distribute materials 
to inform students of available accelerated 
course sequencing.®® This practice would 
likely be more effective when implemented 
in concert with enhanced advising. 


( 40 ) 



Recommendatiori 4 (continued) 


Roadblock 4.2. Adjunct faculty who 
teach developmental education students 
are often disconnected from instructional 
decision-making, professional development 
opportunities, and institutional supports. 

Suggested Approach 4.2. Most 
developmental education courses are taught 
by adjunct faculty,®' who have more limited 
contact with students outside of class, are 
less aware of administrative changes, and are 
less involved in collective faculty decision- 
making, than full-time faculty and staff. 
Without intervention, an over-reliance on 
adjunct faculty to teach developmental 
education courses can limit the capacity of 
institutions to create effective accelerated 
instruction models. When considering the 
implementation of accelerated instructional 
models, adjunct faculty need institutional 
support, professional development, and 
engagement with curriculum development. As 
discussed by Kosiewicz and colleagues (2016), 
the involvement and support of adjunct 
faculty in the design and implementation of 
the accelerated curriculum will strengthen 
their investment and capacity to deliver the 
results anticipated from the new approach. 

Roadblock 4.3. Faculty members 
are accustomed to the status quo 
of existing developmental education 
sequences, structures, and readiness 
levels of students in their classes. 

Suggested Approach 4.3. Colleges 
should provide faculty with professional 
development, opportunities to collaboratively 
plan with other faculty on curriculum 
development, and incentives for faculty 
collaboration.®^ Faculty who currently teach 


college-level courses may resist adaptations 
needed for mainstreaming developmental 
education students in their classes. Smooth 
and successful implementation of the 
changes requires deliberate efforts to 
cultivate champions and support from 
all stakeholders, which takes time and 
persistence. Faculty with expertise in teaching 
college-level content and faculty skilled in 
teaching developmental skills will need to 
partner to maintain instructional rigor for 
college-level mastery while differentiating 
instruction to meet students where they 
are. Communication and collaborative 
decision-making across departments and/ 
or course levels will be necessary. 

Roadblock 4.4. Existing state, 
college, and regional accreditation 
policies may not be well-aligned with 
chosen accelerated course models. 

Suggested Approach 4.4. College- 
level leaders, state policymakers, regional 
accreditors, and discipline-specific 
associations should discuss adjustments 
to any necessary policies and procedures 
that would inhibit students in receiving 
appropriate credit for redesigned courses. 
Transfer and articulation policies may not 
currently support the models identified 
in this guide, and colleges may need to 
work proactively to ensure that new course 
models are recognized and properly 
credited when students transfer. Bracco 
and colleagues (2015) report upon North 
Carolina’s creation of the NC Ready for 
Success Steering Committee as one example 
of how multiple sectors worked together 
to better align policies and practices to 
smooth transitions for students’ success. 
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Recommendation 4 implementation checklist 


Recommendation 4 

Compress or Mainstream Developmental Education with Course Redesign. 

Implementation Checklist^'^ 


Phase I. Evaluation of Current Developmental Education Course Models 


□ Conduct an inventory of the length and scope of current developmental education 
courses. 

□ Investigate the costs associated with the current model. 

□ Investigate the outcomes achieved in the courses under the current model. 

□ Overall 

□ Within different courses 

□ By sub-groups 

□ Work to identify why students are not succeeding in the current courses and 
sequence. 

□ Solicit feedback from faculty. 

□ Solicit feedback from students. 

□ Work to identify why students are successful in the current courses and sequence. 

□ Solicit feedback from faculty. 

□ Solicit feedback from students. 

□ Investigate the effects and lessons learned from previous efforts to compress or 

accelerated developmental education (if any), and use them in planning. 

□ Interview administrators and faculty who participated in previous efforts. 

■ What worked? 

■ What did not? 

■ What do they identify as challenges to success? 

■ What do they suggest as ways to improve? 


Phase II. Planning and Decision-Making for Compressed or Mainstreamed Models 


□ Consider which compression or mainstreaming strategies might work best for your 
college. 
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□ Meet with developmental education faculty to review the strategies outlined in 
this guide. 

□ And, based on students’ needs and college resources, decide which model is 
best for your college. 

□ Decide how your college should choose, sequence, and prioritize strategies and 
solutions. 

□ Determine additional information needed to support students’ successful transi- 
tion to college-level work and how you will determine the success of the strategies 
employed. 


Phase III. Implementation and Evaluation 


□ Cultivate champions of the new approaches, strategies, and models. 

□ Engage faculty in the redesign and obtain faculty support for the new approach. 

□ Offer professional development for faculty to build this team of change agents in 
development education. 

□ Work with developmental education faculty, student support personnel, and core 
academic faculty on the strategies. 

□ Evaluate the outcomes achieved from the mainstreamed or compressed courses 
compared to the traditional course sequences. 

□ Use evaluation data to guide course revisions, resource allocation decisions, advo- 
cacy for policy change to help articulation and transfer as needed, and expansion of 
the accelerated course models. 
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Recommendation 5: Teach students how to become 
self-regulated learners. 

Traditional academic instruction emphasizes learning content. However, many students, 
including those in developmental education, arrive on college campuses without knowing 
much about how they learn and which study strategies might work best for them. 
Teaching students to be more aware of their own learning processes through encouraging 
monitoring and reflection of their learning can help them become better learners. 


Teaching students to be self-regulated 
learners focuses students on the parts of 
the learning process over which they have 
control. Teaching students how to learn more 
effectively can be embedded into existing 
subject-matter coursework. Typically, teaching 
students to become self-regulated learners 
involves demonstrating how to (a) approach a 
task, (b) implement that approach or strategy, 
(c) evaluate how well the approach or strategy 
worked, and (d) decide what to do next.^^ 

Young and Ley (2003, 2001) describe four 
instructional principles and activities to 
support underprepared students to become 
better self- regulated learners: I) prepare and 
structure the learning environment to help 
students cope with distractions, 2) organize 
and transform instructional materials to 
support self-regulated learning, 3) work 
with students to keep records and monitor 
progress, and 4) work with students to 
evaluate performance against a standard. 

Summary of evidence: 

Minimal Evidence 

The panel judged the level of evidence 
supporting students in regulating 
their own learning as minimal. 

One study met WWC standards without 
reservations.®^ Zimmerman et al. (2011) 
observed statistically significant effects on 
three measures of progress in developmental 
education, including the best measure: the 
proportion of students who passed the 
college-level course in which remediation 
was needed (in this case, math). In the 
intervention group, 25% of students passed- 


college level math; only 13% of students in 
the comparison group did. See Appendix 
D Table 5 for more detail on this study. 

How to carry out the 
recommendation 

Again, teaching students how to become 
self-regulated learners involves introducing 
students to the process of (a) approaching 
a task, (b) implementing that approach 
or strategy, (c) evaluating how well the 
approach or strategy worked, and (d) deciding 
what to do next. Figure 5.1. provides an 
overview of the self-regulated learning 
process for instructors and students. 

1. Encourage students to improve 
their estimation of their current 
capabiiities. When students plan which 
strategies to use in a learning task, they 
have to make an accurate judgment 
of their capabilities and resources. 
Completing tasks in which students 
have been previously successful will 
likely take less forethought, because 
they already know strategies that work 
in those learning situations. When 
students encounter new learning tasks 
or situations, they will have to use their 
own judgment of how they performed 
previously on similar tasks and which 
strategies might be relevant. However, 
struggling learners are often poor judges 
of their own knowledge and capabilities.®® 
Monitoring and reflecting on the learning 
process allows students to practice 
judging their abilities, performing a task, 
and getting feedback to improve their 
judgment of their own knowledge and 
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capabilities. For example, instructors 
can prompt students to reflect on what 
they think they know and can do while 
doing an academic task. After students 
receive feedback on the task, instructors 
can ask students to reflect on whether 
they were as able as they thought. 
Zimmerman et al. (2011) observed that 
after students in developmental education 
courses practiced this type of structured 
reflection, they were better judges of their 
own abilities than students who did not 
regularly participate in the reflection. 

2. Integrate monitoring and reflecting 
on the iearning process with 
coursework or course content. 

Students can learn to be better learners 
while learning course content. Instructors 
can structure their courses to support 
this by integrating into the course 
expectations and assignments student 
planning, reflection, and evaluation of 
the learning process.'°° Instructors can 
prompt students at the start or end of 
assignments to aid students in monitoring 
and reflecting on their learning. These 
prompts can be steps that students 
have to go through to complete the 
assignment or process questions to 
direct students to evaluate their learning 
approach and determine what to do on 
similar tasks in the future. For example, 
for more general studying behaviors, 
instructors can ask questions, such as: 
“Flow do you take notes in class? Do you 
review your notes? When? Flow? Do you 
stop periodically and check to see if you 
are understanding the material?”’®' 

The intervention in Zimmerman et 
al. (2011) gave students a chance to 
learn from their errors on quizzes by 
completing a self-reflection form in which 
they had to compare their estimates of 
ability with their quiz score, explain why 
they got the quiz item wrong, select a 
different strategy to try in the future. 


state their confidence that that strategy 
would be effective, and complete a 
similar problem to the one they answered 
incorrectly on the quiz. Teaching students 
to improve their learning process can 
also occur in a standalone course that 
makes linkages to content in courses that 
students are concurrently attending. See 
Figures 5.2 through 5.7 for examples. 

3. Present students with examples 
and models of how to monitor and 
reflect in the course subject matter. 

Instructors can model monitoring and 
reflecting through making their thinking 
explicit in classroom discussions and 
lectures so that students have a model 
for how to approach an academic task. 

For example, an instructor might explain 
to students how to determine what is 
important in a reading passage. Or, 
instructors can ask students to be explicit 
about their monitoring and reflecting in 
class discussions. Student responses in 
discussion or homework problems can 
be used in the same way as instructor 
modeling of monitoring and reflecting.'®^ 

In addition, these examples may be more 
accessible for students because they may 
be more similar to their own thinking. 

For example, in Zimmerman et al. (2011), 
“the instructor encourage[d] students to 
go to the board to demonstrate problem 
solving and error detection strategies, 
and . . . the instructor encourages 
students to verbalize error detection and 
problem solving strategies while working 
through practice problems” (p. 146). 

Potential roadblocks and 
suggested approaches 

Roadblock 5.1. Some faculty may resist 
teaching developmental education students 
to regulate their learning because of scarce 
time and resources in the classroom. 
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Suggested Approach 5.1. Assure faculty 
that teaching students to be better self- 
regulated learners leads them to be more 
effective and efficient learners in the long 
term. Hudesman et al. (2013a) noted that 
new instructors are often resistant because 
of the extra time required to implement self- 
regulated learning programs. Experienced 
instructors may also have these concerns, 
particularly since adding more content or 
lessons to a course will necessitate the use 
of class time. However, as suggested by 
Hudesman et al. (2013a), after initial lags in 
learning, students may actually learn more 
efficiently once they become more effective 
self-regulated learners. Also, once students 
become more self-directed, the need for 
faculty intervention may fade.’°^ Hudesman 
et al. (2013a) also found that instructors may 
find it useful to automate administration and 
scoring of quizzes with tablet PCs to create 
time efficiencies and engage students with 
online feedback on quizzes and assignments. 

Roadblock 5.2. Building students’ 
learning skills may not fit into 
faculty goals for a given course. 

Suggested Approach 5.2. Involve faculty as 
meaningful partners in integrating practices 
to teach self-regulated learning. There are 
many reasons why faculty members may not 
see building learning skills as appropriate for 
their course. For example, they may assume 
that students already have these skills or 
that they should be learning them elsewhere. 
Given that the specific strategies chosen are 
likely less important than finding strategies 
that faculty can implement well and believe 
will be effective, there is room for faculty 
choice in the specific strategies implemented, 
which may facilitate faculty investment. 

Roadblock 5.3. Students may resist 
monitoring and evaluating their 
learning process because they fee! 
it is unfamiliar, cumbersome, time- 
consuming, irrelevant, or useless. 


learning processes and give students time 
and space to internalize these practices. 

As postsecondary learners, students in 
developmental coursework have a long 
history of learning which, oftentimes, 
was less productive than it could have 
been. Instructors may find that students’ 
engagement in self-regulated learning 
processes in academic coursework increases 
after engaging students in applying these 
techniques to career objectives and personal 
goals such as improved relationships, health 
and athletics, weight loss, and others. 

Students may be set in their ways about 
what works for their own learning. Or, 
students may not have considered or used 
more effective strategies that require them 
to monitor and reflect on their learning 
process. Also, thinking about the learning 
process while they are learning may feel 
more difficult to students, and the pace of 
learning can seem slower. Instructors can 
present self-regulated learning instruction 
as a way for students to become better 
learners, which will help them in their current 
and future classes and careers. Instructors 
can discuss how they use these strategies in 
their own learning and how other students 
have used these strategies to become 
more successful learners. Instructors can 
also model strategies in the classroom. 

If instructors can give students time, space, 
and instruction to support them in becoming 
self-regulated learners, instructors may 
overcome the initial negative reaction to 
the novelty of the practices. For example, 
Hudesman et al. (2013b) noted that students, 
after being exposed to a self-regulated 
learning program, said the program created a 
“positive educational experience” regardless 
of whether they passed the course.'®"* 

Roadblock 5.4. Instructors may not be 
skilled self- regulated learners themselves. 

Suggested Approach 5.4. Provide 
instructors with materials and training so 
they can put the skills they are teaching 
developmental education students into 


Suggested Approach 5.3. Discuss the 
relevance of reflecting on students’ own 
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practice. Instructors may not use these 
practices regularly in their own learning. Or, 
they may be unfamiliar with terminology 
or steps for implementation. Like students, 
instructors can become more skilled in self- 
regulated learning through direct instruction 
and practice. In a supplemental report 
to Zimmerman et al. (2011), Hudesman 
and colleagues (2014) described how 
instructors in the intervention attended 
a 2-day onsite training program to learn 
how to implement self-regulated learning 


practices in their classes.'^ Over the 
semester and after the training program 
ended, instructors completed logs covering 
classroom experiences and how they 
applied self-regulated learning. The logs 
gave instructors practice participating 
in the self-regulated learning process, 
because they asked instructors to reflect 
on their goals for the class period and 
evaluate how effective their strategies 
were implemented.'°® Instructors received 
feedback on their logs from program staff. 
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Figure 5.1. Overview of the self-regulated learning process'°^ 


INSTRUCTORS 



STUDENTS 


PRACTICE 
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Figure 5.2. Overview of the self-regulated learning process 


Integration of Self-Regulated Learning Throughout a Course 

The following instructional tools and examples illustrate how to integrate the teaching 
of self-regulated learning within quizzes. Self-regulated learning strategies — such as 
student predictions, assessment of confidence, reflections on performance — should be 
embedded in homework, tests, and other course assignments. Students should also be 
encouraged to apply them in other courses. 
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Figure 5.3. Sample self-regulated learning quiz in mathematics 


SAMPLE 

Self-Regulated Learning Quiz in Mathematics 

The following is an example of a quiz that helps students focus on their self-efficacy to 

do class tasks. Over time, comparing the before and after judgments can help students to 

improve their Judgment of their capabilities and knowledge of class topics and tasks. 

Name 

Date Quiz # 

Predicted Score: 

Preoaration Time: 

mins. 

BEFORE solving each 

REMEMBER! 

AFTER you have solved 

problem, how confident 

Show all your work. 

each problem, how con- 

are you that you can 

Simplify all your answers. 

fident are you that you 

solve it correctly? 


solved it correctly? 

0% 

1. Factor completely: 

0% 

25% 

lOx^y^ -1- 4xy^ - 2y = 

25% 

50% 


50% 

75% 


75% 

100% 


100% 

0% 

2. Solve: 

0% 

25% 

8a^b^- ]2a^b^c+ 4ab^ 

25% 

50% 

Aab^ 

50% 

75% 


75% 

100% 


100% 

0% 


0% 

25% 

3. Express in scientific notation: 

25% 

50% 

a) 6700000 b) 0.000015 

50% 

75% 


75% 

100% 


100% 

0% 


0% 

25% 

4. Compute and express in 

25% 

50% 

scientific notation: 

50% 

75% 

(3.6 X 10=)(6 X 10")/12 X 10= 

75% 

100% 


100% 

0% 


0% 

25% 

5. Solve: (5x - 3)= 

25% 

50% 


50% 

75% 


75% 

100% 


100% 
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Figure 5.4. Sample quiz reflection and revision sheet in mathematics 


SAMPLE 

Quiz Reflection and Revision Sheet in Mathematics"^^ 


After students receive feedback on their quiz performance, the Revision Sheet asks 
them to think about whether the strategies that they chose to use were appropriate, 
which strategy to use on an item that they got incorrect, and whether they feel confi- 
dent that the new strategy will be effective. 


SRL Math Revision Sheet, Quiz # Item # 

Student: Instructor: 

Date: 


Now that you have received your corrected quiz, you have the opportunity to improve 
your score. Complete all sections thoroughly and thoughtfully. Use a separate revi- 
sion sheet for each new problem. 


8 PTS 

PLAN IT 

1. a. How much time did you spend studying for this topic area? 

b. How many practice problems did you do in this topic area in preparation for 
this quiz? 

(circle one) 0-5/5-10/1 0-i- 

c. What did you do to prepare for this quiz? (use study strategy list to answer the 
question) 

2. After you solved this problem, was your confidence rating too high (i.e., 4 or 5)? 
yes / no 

3. Explain which strategies or processes went wrong on the quiz problem. 


8 PTS 
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PRACTICE IT 

4. Now redo the original quiz problem and write the strategy you are using on the 
right. 

Problem Strategy 


5. How confident are you now that you can correctly solve a similar item? 

1 Definitely not confident 

2 Not confident 

3 Undecided 

4 Confident 

5 Very confident 


6. Now use the strategy to solve a similar problem. 

Problem Strategy 


4 PTS 


EVALUATE IT 

7. How confident are you now that you can correctly solve a similar problem on a 
quiz or test in the future? 

1 Definitely not confident 

2 Not confident 

3 Undecided 

4 Confident 

5 Very confident 
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Figure 5.5. Paired discussion among students — evaluation of the quiz format 


Paired Discussion among Students 
Evaluation of the Quiz Format''” 


This activity links with the quiz reflections that students are assigned. The partner 
format helps students reflect on how they can be more engaged (“active”) learners 
and focus on the process of learning rather than their own unchangeable or innate 
abilities. 


Sample Discussion Questions about the Self-Regulated Learning Quiz Format 

Find a partner and discuss your reaction to the Self-Regulated Learning Quiz format, 
using the following questions. 

1. What are some of your initial reactions to this quiz format? 

2. How can this quiz format help you become a more active learner? 

3. What is the relationship between the time you spend preparing for the quiz and 
your quiz score? How can you check out this relationship? 

4. Why is it useful to judge how well you think you did on a problem before AND 
after you complete the problem? 
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Figure 5.6. Guided self-reflection about the course 


Guided Self-Reflection About the Course'" 


This activity helps students to reflect on their growth as learners throughout the 
course. The questions ask students to discuss specific strategies or examples of being 
an “active learner” as well as how they’ve become more effective learners because of 
the course. 


Sample Self-Reflection Questions About the Course 

Think about the semester so far, and answer the following questions: 

1. What are two strategies that work for you in math class that you would recom- 
mend to your classmates? 

2. What has changed about you as a math student since the first week of class? 

3. What are two examples of why you are more of an active learner than you were 
the first week of class? 

4. What is one question you have for your classmates or instructor? 
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Figure 5.7. Discussion questions for applying self regulated learning skills to 
other courses 


Discussion Questions for 

Applying Self- Regulated Learning Skills to Other Courses"^ 


This activity encourages students to transfer the self-regulated skills that they have 
learned to their other coursework. 


Sample End-of-Course Discussion Questions (for students on how to trans- 
late these strategies to other courses) 

During the discussion about monitoring, take notes on significant points and ideas 
you want to remember. 

1. What are examples of monitoring tools you have noticed other people (instructors, 
friends, etc.) using? 

2. What is the relationship between self-monitoring and academic success? 

3. What makes an effective self-monitoring tool? 

4. How can your self-monitoring tool affect whether or not you reach your goal? 

5. What new self-monitoring tool would you like to try? Why do you want to try that 
self-monitoring tool? 
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Recommendation 5 

Teach Students How to Become Self-Regulated Learners 

Implementation Checklist"^ 


Planning 


□ Consider how to integrate the teaching of self-regulated learning skills into your 
course structure and syllabus. 

□ Adapt course quizzes, assignments, tests, and homework to incorporate self-regu- 
lated learning components. 

□ Throughout the course, include time and space for students to make plans for doing 
their assignments and evaluate their follow-through on the plan they created. 

□ Finalize a plan for course curriculum, instruction, and assessment, as well as ways 
to evaluate the effectiveness of your plan in your course (and courses in your 
department, as applicable). 

□ Make predictions about how well your students will do in the course; and document 
the work performed to prepare for the course and your level of confidence in how 
effective the self-regulated learning process will be. 


Implementation 


□ Motivate and introduce students to self-regulated learning strategies as a way to 
improve their learning. 

□ Guide students to understand that they have some control over their learning pro- 
cess and they can become active, self-regulated learners not just in this class, but in 
all their classes. 

Strategies include: 

□ Ask students to evaluate their pre-existing learning processes and compare 
these to what self-regulated learners do. 

□ Ask students to share strategies that were effective and ineffective. 

□ Provide assignments to scaffold the self-regulated learning process. 

For example, in quizzes: 

□ Ask students to predict their quiz score. 
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□ Have students record the time they spent preparing for the quiz, and their level 
of confidence in answering correctly. 

□ After they have completed the quiz, ask students whether they believe they 
succeeded and whether their predictions were correct. 

□ Invite students to reflect upon why they succeeded or not, and write down 
plans for what they will do for future assignments. 

□ Integrate self-regulated learning instruction with course content, so that students 
have a disciplinary context in which to apply their self-regulated learning skills. 

□ In communication with students, reinforce how assignments will help them become 
more active learners so that they can transfer these skills to other coursework. 

Strategies include: 

□ Provide reflection points for students to pause and plan their learning or to 
evaluate their learning and their learning plan. 

□ Establish classroom norms that allow students to support each other in becom- 
ing better learners. For example, let students know that asking for help is not 
an indication of failure; it is an indication that they are honestly evaluating their 
own learning process and trying to improve. 

□ Teach students to set achievable goals. Student learning goals should be quan- 
tifiable, have a time limit, and have a strategy connected with them. 

□ Have students record changes to goals and plans so that they can look back, 
evaluate, and improve their goal-setting skills. 

□ Emphasize in discussion and assignments the importance of learning from 
feedback to improve future learning. 

□ Show students ways of organizing or viewing feedback to make the feedback 
more useful. For example, students may summarize written feedback across 
assignments or use online tools to graph changes in feedback. 

□ Have students make predictions about both short-term (i.e., assignments) and long- 
term (i.e., course) learning performance. 

□ Have students carry out an academic plan and evaluate how well the plan worked. 

□ Work with other instructors to reinforce self-regulated learning skills in their 
courses, so that students can apply these skills in multiple academic courses. 

□ Guide students to apply self-regulated learning processes to other courses, even 
when instructors do not explicitly integrate them. 
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Evaluation 


□ Evaluate how well self-regulated learning processes were integrated into course 
content and whether your predictions of student performance were achieved with 
the methods implemented. 

□ Based on analysis of course evaluations and student performance in the course, 
modify curriculum, instruction, and assessment practices as needed. 

□ Write a plan for the next course’s curriculum, instruction, and assessment, as well as 
make any needed changes to course evaluation or departmental planning. 
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Recommendation 6: Implement comprehensive, 
integrated, and long-lasting support programs. 

Some institutions have implemented comprehensive and integrated support programs that 
incorporate a variety of components with the goal of improving student educational outcomes. 

While many colleges offer multiple supports to their students, what differentiates this practice 
from business as usual is the intentional focus on integrating these supports and incentivizing 
participation in the long term. One example is the City University of New York’s (CUNY) Accelerated 
Study in Associate Programs (ASAP). ASAP provides a “comprehensive array of services and supports 
over a three-year period to help more students graduate and to help them graduate sooner.”''"^ 


Summary of evidence: 

Moderate Evidence 

The panel has assigned a moderate 
evidence rating for this practice. 

The rating is based on the expert judgment 
of the panel and on one large randomized 
controlled trial of 896 low-income students 
with developmental education needs’'^ that 
meets WWC standards without reservations. 

This study revealed statistically significant 
effects on its primary outcomes; many of 
these effect sizes were large in magnitude. 
Specifically, students who participated in 
ASAP were more likely to complete their 
developmental education requirements (75% 
completion rate in the ASAP group versus 57% 
in the comparison group); earn better grades 
(about 69% of the ASAP group and 60% of the 
comparison group earned at least a 2.0 GPA); 
earn more college credits (after 6 semesters, 
the ASAP group accumulated 43 credits and 
the comparison group earned 35 credits); earn 
a degree (usually an associate’s degree) at the 
study’s longest follow-up date (by the sixth 
semester, 40% of the ASAP group and 22% of 
the comparison group had earned a degree); 
and enroll at a four-year college within 3 
years of entering the program (transfer rates 


were 25% in the ASAP group and 17% in the 
comparison group). Figure 6.1. and Appendix 
D Table 6 summarize these impacts. 

Of the interventions studied for this guide, 
ASAP was the largest in terms of scale 
and scope. As a result of the expansive 
scope of the ASAP intervention, the 
panel observed greater improvement 
across more outcomes than in any of the 
other practices in this guide. While other 
practices received a moderate rating from 
the panel, this practice — implementing 
comprehensive, integrated, and long-lasting 
support programs — showed the largest 
increases in effects on college success 
outcomes, including degree attainment. 


“While other practices received a 
moderate rating from the panel, 
this practice — implementing 
comprehensive, integrated, and 
long-lasting support programs — 
showed the largest increases 
in effects on college success 
outcomes, including degree 
attainment." 
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Figure 6.1. Postsecondary impacts of Acceie rated Study in Associate Programs 


Postsecondary Impacts of 
Accelerated Study in Associate Programs (ASAP)"^ 



Enrolled in 
postsecondary 
education 
(semester 1) 


Cumulative Earned a degree 

college-level credits within 3 yrs. 

within 3 yrs. 


Completed Achieved greater 

developmental than 2.0 CPA 

education requirements (semester 1) 

within 3 yrs. 


Transferred to 
four-year institution 
within 3 yrs. 


OUTCOME MEASURES 














Recommendation 6 (continued) 


Figure 6.2. Illustrative example of comprehensive, integrated, and long-lasting supports 


Illustrative Example of 

Comprehensive, Integrated, and Long-Lasting Supports 



Student 

Services 

(Enhanced 
Advising, 
Tutoring, Career 
Services) 


Requirements 
& Messaging 

(Full-Time 
Enrollment, Take 
Developmental 
Courses Early, 
Graduate in 3 
Years) 


Cohort-based 

Classes 

(Student Success 
Seminar, Block- 
Scheduled 
Classes, Early 
Registration) 


Financial 

Supports 

(Tuition Waiver, 
Free Public 
Transportation 
Cards, Free 
Textbooks) 


Long-Lasting Support 


Strong, Collaborative Organizational Structure that Uses Data to Inform Practice 
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How to carry out the 
recommendation 

Many students in developmental education 
face a range of challenges."^ To ensure 
that they can succeed, institutional leaders 
should consider programs that combine 
student services, specific requirements, 
scheduling that encourages student cohorts, 
and financial supports. The impact of 
combining these supports appears to be 
more than simply providing additional 
help and instead creates a clear path to 
students’ success by systemically eliminating 
institutional barriers that create obstacles 
to students’ progress and attainment. 

The ASAP intervention is one example 
of how to carry out comprehensive, 
integrated support programs for 
developmental education students. 

The primary purpose of ASAP was to 
accelerate progress through college so 
that students could graduate in 3 years. 

ASAP required full-time attendance and 
provided students with services (such as 
comprehensive advisement, tutoring, and 
career assistance) and financial support (such 
as tuition waivers, free textbooks, and free 
public transportation passes). To ensure 
student schedules that facilitated accelerated 
study, ASAP offered and encouraged students 
to take blocked or linked courses for the first 
year and a student success course for the first 
few semesters. ASAP students were able to 
register for courses early, which helped them 
to design a schedule suitable to their needs. 
ASAP students were encouraged to take their 
developmental courses early and to graduate 
within 3 years. To successfully implement all 
of the programmatic components, a strong, 
collaborative organizational structure was 
developed. Figure 6.2. provides an overview 
of the components of this program. 

1. Develop a strong, collaborative 
organizational structure that uses 
data to inform practice. Attention 


to the oversight and coordination of a 
comprehensive and integrated support 
program is critical. A large number of 
people and units will need to be involved 
in fairly constant communication for 
a program like this to be successful. 
Leadership at different levels is also 
necessary. For example, ASAP was 
implemented as a partnership between 
the CUNY Office of Academic Affairs 
and three participating community 
colleges. This required multiple layers of 
organizational structure and high levels 
of collaboration. The Office of Academic 
Affairs provided fiscal and programmatic 
oversight while the individual colleges 
established ASAP teams that operated 
the programs at their sites and provided 
direct services to the students. The 
central office staff and college ASAP 
directors met monthly and communicated 
frequently between meetings."® 

It is also important to use data to 
inform program structure, monitor 
program implementation and usage, 
and evaluate effectiveness. ASAP 
implementers used data to: 

■ Develop the program structure 
based on motivating factors and 
needs of the student population. A 
comprehensive, integrated, long- 
lasting support program may look 
quite different from one location to 
another because students in different 
colleges are motivated by different 
incentives and have different needs. 
For example, in New York City, 
many students consider MetroCards 
to be valuable, given the expense, 
efficiency, and popularity of public 
transportation. Thus, MetroCards 
were given as an incentive to fulfill 
program requirements, such as going 
to advising appointments. In other 
locations, gift cards for gasoline 
may be a comparable incentive. 
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■ Track the students as they go 
through the program and meet the 
various requirements. Students 
are required to enroll full time and 
attend advising, tutoring, career 
services, and the ASAP seminar. 

These activities need to be tracked to 
identify which students are eligible 
to receive their monetary incentives 
and remain in the program. To keep 
track of this information, each ASAP 
college recorded information about 
contact with participating students 

in a centralized data management 
system. Contacts with advising and 
career services were logged monthly, 
while ASAP seminar participation and 
tutoring visits were tracked weekly. 
CUNY central office evaluation staff 
analyzed program data monthly, and 
shared reports with the colleges. Key 
program staff and administrators 
then used the data to manage and 
modify the program, so that program 
benchmarks would be met. If data at 
a college showed, for example, that 
students were not meeting the target 
to meet with their advisor twice a 
month, the CUNY central staff and 
the college’s program director would 
enact solutions to resolve this issue. 

■ Assess program effectiveness. The 
central office conducted periodic 
surveys of ASAP students and held 
student focus groups about their 
perceptions of and satisfaction 
with the program. An impact 
evaluation was also conducted. 

2. Clearly communicate expectations 
to students. By way of example, ASAP 
communicated its goals of full-time 
enrollment, taking developmental courses 
early, and graduating quickly through 
written materials, program orientation, 
and discussions with advisers."® The 
messaging began immediately as 


students entered ASAP and continued 

throughout the program. The primary 

messages students received were to: 

■ Enroll full time and take intersession 
courses. Full-time enrollment was 
required for the ASAP intervention 
in each fall and spring semester 
that students were enrolled in the 
program. Advisors also encouraged 
students to take courses in the 6-week 
intercessions. Although it is still 
unknown which program components 
resulted in ASAP’s positive effects, 
the researchers speculated that the 
full-time enrollment requirements, 
coupled with the multiple supports to 
facilitate that enrollment, appear to be 
“central to the program’s success’’.'^® 

'2' Colleges may also be able to be 
flexible when defining “full time.” 

For example, if full time is defined 
as 12 credit hours per semester for 
an undergraduate student, it could 
be redefined to 24 credit hours in 
an academic year (perhaps with a 
minimum of six credit hours in the 
fall and spring semesters). Doing 
so would allow students taking 
courses during intersessions and in 
the summer to count these toward 
their full time enrollment status, 
thereby both encouraging a more 
continuous enrollment pattern 
and providing some degree of 
flexibility for students who cannot 
manage four courses at a time. 

■ Take developmental courses early. 
Although ASAP did not incorporate 
an accelerated developmental 
coursework sequence, it promoted 
taking developmental courses early, 
mostly through conversations about 
course planning and registration 
with advisors. Developmental 
courses for ASAP students were 
often part of the linked or blocked 
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courses offered to students 
during the first 2 semesters.'22 

■ Graduate quickly. ASAP promoted 
a messaging strategy that students 
“should and can graduate within three 
years”.'^^This message was promoted 
during program orientation, through 
written materials, and during advising 
sessions from the beginning through 
the end of the program. Taking 
advantage of winter and summer 
intersessions also helped students 

to graduate within three years. 

3. Design course enrollment 
strategically. In ASAP, program 
developers designed course enrollment 
structures to group ASAP students 
together and allow ASAP students to 
maximize the likelihood of maintaining 
a full-time schedule. This included: 

■ Blocked or linked courses. ASAP 
students may enroll in linked courses 
in their first year. These linked courses 
include two or more classes grouped 
together with seats reserved for ASAP 
students.'^"' The purpose of linking 
was to create a learning community 
for students, which are based on 

the belief that active learning in a 
community based setting can improve 
academic outcomes by increasing 
academic and social integration.'^^ 

■ Student success course. ASAP Students 
enroll in an ASAP seminar as a cohort 
during their first few semesters.'^® 

This seminar was a noncredit course 
covering topics such as problem 
solving, stress management, study 
skills, and campus information. The 
seminar also helped with career and 
transfer planning and facilitated 
discussions on topics such as leadership 
and community involvement. 

■ Early registration. ASAP students can 
register for courses early,'^'' which 
gives students who work full time (or 


have other out-of-school responsibilities) 
the best chance to create a course 
schedule that fits their lives. 

4. Decide which student support 
services wiii be provided and how 
to ensure that students participate. 

There are a number of different types 
of student support services that can 
be offered to developmental education 
students. In ASAP, participation in 
enhanced advising, tutoring, and 
career services was mandatory. There 
were also some voluntary services. 

Students were provided with: 

■ Enhanced comprehensive advising on 
academic, social, and interpersonal 
issues by ASAP advisors. The student- 
to-advisor ratio was between 60:1 
and 80:1, and ASAP participants met 
with their advisors very frequently (an 
average of 21 times in the first semester 
and 17 times in the second semester; 
Scrivener et al., 2015). The ASAP 
advisors worked specifically with ASAP 
students and had an ASAP supervisor 
who helped to define their role. 

■ Required tutoring during their 
developmental courses and during 
any semesters they were on academic 
probation. Each of the participating 
colleges hired tutors dedicated 

to ASAP students. Tutors were 
generally former students at four- 
year colleges or adjunct faculty.'^® 

■ Required meetings with an ASAP- 
dedicated career and employment 
specialist. Topics include balancing 
work and school, finding scholarships, 
finding part-time work, networking, 
writing resumes, and aligning college 
majors with career paths. Participants 
engaged in an average of five meetings 
in the first semester and four meetings 
in the second semester. Students 
continued to visit with career and 
employment specialists at “comparatively 
high rates’’’^® in the second and third 
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years. Visits began decreasing in 
the fourth and fifth semesters. The 
career and employment specialist also 
hosted career fairs and job talks. 

Voluntary services were also provided. 

For example, ASAP participants could 
meet with a social work intern one-on- 
one as needed. Although this service had 
much potential, it was staffed by rotating 
interns, and having a new intern each 
semester was identified as a challenge to 
the popularity and effectiveness of this 
offering.'^° Additionally, a small number 
of students were selected to participate 
in the ASAP Student Leadership Program, 
which focused on public speaking skills, 
teamwork, diversity, and advocacy. 

5. Provide performance-based 
monetary incentives and other 
bnanciai supports. In exchange for 
program participation, ASAP provided 
three primary monetary incentives: 1) 
financial assistance with textbooks, 2) a 
tuition waiver that covered the difference 
between a student’s tuition and fees 
and financial aid, and 3) a monthly 
transportation pass. The requirement for 
ASAP participants to receive the program’s 
financial benefits was enrollment in at 
least 12 credits or more per term. A 3.0 
GPA or higher was required to receive 
advisors’ discretionary funds for covering 
students’ winter or summer coursework. 

To address course-related financial 
concerns, ASAP students received 
free textbooks for all courses. At the 
end of the semester, students were 
responsible for returning the books. 

ASAP staff estimated the textbook 
cost for a full-time student to be 
approximately $270 per semester. 

ASAP waived the difference between a 
student’s financial aid package and their 
tuition and fees. Financial aid offices 
and college ASAP directors worked 
together to ensure that the tuition waiver 


was applied to accounts internally and 
not paid directly to students. Only 9% 
of ASAP students needed this extra 
financial assistance during the first 
semester of study participation; during 
the third year of the study, 3-5% of 
the participants received this waiver. 
Among students receiving a waiver, 
the average amount was $719.'^' 

Students in the ASAP program were 
given free monthly MetroCards for 
public transportation, contingent on 
participation in program requirements, 
such as advising, tutoring, and full-time 
enrollment. Among the ASAP participants, 
93% reported that they had received 
financial assistance with transportation 
in the first year of the study, and the 
vast majority of ASAP students received 
their MetroCard in most or all of the 
months when they were taking classes. 
The cost of the transportation passes 
increased during the 3-year evaluation. 

By the end of the follow-up period in 
2013, this cost was $112 per month.'^^ 

ASAP provided a comprehensive, integrated 
package of student services, monetary 
incentives, linked courses, an ASAP seminar, 
and other supports. Furthermore, CUNY sent 
consistent, strong messages to ASAP students 
to enroll full time, take their developmental 
classes early, and graduate within three 
years. The available evidence does not 
show which of these program components, 
or which combination of components, 
were responsible for the success of the 
program. Flowever, based on the relatively 
large effects achieved by ASAP relative to 
the effects of an individual program or a 
combination of supports implemented for 
a short time, the expert panel recommends 
adopting a systematically integrated set 
of interventions over multi-year period 
over any single intervention in isolation. 

For additional guidance, CUNY published a 
resource guide for college administrators. 
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faculty, and other key stakeholders interested 
in instituting a program such as ASAP at 
their own institution. The resource guide 
shares information about program structure, 
program components and implementation, 
program evaluation and management, as 
well as sample planning templates.'^^ 

Potential roadblocks and 
suggested approaches 

Roadblock 6.1. The college is on a 
limited budget and cannot afford to 
provide such comprehensive services. 

Suggested Approach 6.1. Focus on the 
long-term benefits of cost per degree from 
this type of program. Comprehensive and 
integrated support programs are expensive to 
implement compared with business-as-usual 
models. Over 3 years of implementation. 
Scrivener et al. (2015) estimated the CUNY 
ASAP program cost (which included the 
base cost of the program plus indirect 
costs attributable to the program) per ASAP 
student per year to be about $5,400 more 
than the cost for a student who received 
the usual college services. However, the 
base operating cost per student for ASAP 
at CUNY fell about by about 20% after 
the first few years of implementation, to 
about $3,900 additional cost, per student 
per year, compared to students who 
received typical college services.'^"^ 

Even though it is more costly at the front 
end to implement comprehensive and 
integrated support programs, the benefits 
can outweigh the costs. For example, the 
CUNY ASAP program generated more 
graduates than the business-as-usual 
comparison (40% versus 22%, respectively). 
And, although beyond the scope of this 
guide, a benefit-cost study suggested 
that the cost per degree was lower.'^^ 


Roadblock 6.2. Sustaining student 
interest and progress through 
the summer is challenging. 

Suggested Approach 6.2. Promote and 
support students’ year-round enrollment 
whenever possible. To the extent possible, 
year-round enrollment should be a priority 
for underprepared college students. This 
roadblock may also be alleviated by providing 
student assistance with employment, 
providing on-campus employment, 
addressing students’ financial aid challenges 
for summer courses, and offering other 
student supports (e.g., child care). 

Roadblock 6.3. Turnover in leadership 
and objections to change among staff can 
hinder the implementation of the program. 

Suggested Approach 6.3. For a 

comprehensive, integrated, and long- 
lasting program to be successful, provide 
and support leadership, coordination, and 
collaboration at all levels. Leadership and staff 
turnover during a long-term implementation 
process presents a formidable challenge. One 
strategy for ensuring collective responsibility 
of long-range implementation of the program 
is to engage stakeholders throughout the 
institution in the coordinated design of 
the program and ongoing learning and 
modifications. Program developers must 
provide a structure for collaboration and 
support the participation of key individuals 
involved in each facet of the programming. 
Given the positive outcomes for the ASAP 
program, available evidence may be 
effective at convincing staff to participate 
and make changes to existing practices and 
policies. Champions should be cultivated 
from all the important sectors and levels 
to ensure adaptive implementation, as 
leadership and staffing changes occur. 

With structures for collaborative solution- 
finding, discussion of existing data on current 
programs’ impact on student outcomes. 
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and broad participation in learning from 
existing evidence, champions at all levels 
can be effectively created and nurtured. 

Roadblock 6.4. Many students placed 
in developmental education are part- 
time, so requiring full-time enrollment 
would exclude that population. 

Suggested Approach 6.4. Depending on 
the proportion of your institution’s full-time 
and part-time enrollment, consider designing 
comprehensive, integrated, and long- 
lasting programs for both full-time enrolled 
students and part-time enrolled students, 
while incentivizing students to enroll full 
time. About 40% of students enrolled in U.S. 
community colleges are enrolled full time,'^® 


and, as a sector, community colleges serve 
the largest share of students underprepared 
for college. Although some researchers 
question whether a full-time attendance 
requirement benefits all students in 2-year 
programs,'^^ most research does find a 
relationship between full-time enrollment 
and positive outcomes for college students, 
especially when implemented with sufficient 
planning and supports for the students — 
and not in isolation. Some part-time 
students may be incentivized to enroll full 
time to have the benefits of an ASAP-like 
program, while other part-time students 
may not be able to alter their enrollment 
and yet benefit from a modified ASAP 
program geared toward part-time students. 
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Recommendation 6 implementation planning checklist 


Recommendation 6 

Implement Comprehensive, Integrated, and Long-lasting Support Programs. 

Implementation Checklist'^^ 


Enrollment Targets and Eligibility 


□ What is your current enrollment by major with full-time and part-time distributions? 

□ What are your current 1-year retention and 3-year graduation rates? If possible, 
disaggregate by majors and full-time vs. part-time status at time of entry. 

□ What is the developmental education profile for your most recently admitted first- 
year student cohort (e.g., by skills area and level of need) and proficiency status 
after two semesters? 

□ What is the percentage of students who transfer to 4-year colleges with and without 
a degree by major? 

□ Which students do you plan to target? Consider majors as well as enrollment status 
(e.g., first-time freshmen, transfer students, continuing students, etc.). 

□ Which eligibility criteria will you use with regard to financial need, credits accumu- 
lated, developmental needs, etc.? 

□ What are your program enrollment targets? 


Financial Incentives and Resources 


□ What financial incentives can you offer to students (e.g., textbooks, transportation 
vouchers/gas cards, childcare, tuition and fee waivers, etc.), and how will this affect 
your annual program budget and enrollment targets? 

□ Will you be able to provide any funds for students to take courses in winter and 
summer terms? 


Staffing 


□ How will you identify dedicated program staff to administer the program? What will 
your plan be for hiring new staff? 

□ What will your advisor-to-student ratio be? 

□ Who will create and lead staff training, and is there a plan for ongoing professional 
development? 
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College Integration 


□ Where will the program be housed, and what will the reporting structure be? 

□ What will your communication plan be to facilitate collaboration with other college 
units? (i.e., admissions, financial aid, testing, registrar, learning support center(s), 
bookstore, bursar, budget office, academic departments, adult/continuing educa- 
tion, etc. 

□ What are three to five strategies you will use to ensure that program staff members 
are working collaboratively with other college units and that program services are 
integrated into existing college systems? 

□ Is there interest in faculty directly supporting students? If so, for what type of stu- 
dent? What types of supports could be provided by faculty? 

□ Are there professional development opportunities for faculty working closely with 
your program, especially in the areas of advisement? 

□ Which systems will encourage faculty, especially those teaching blocked courses, to 
provide feedback on student progress? 

□ Are there opportunities for faculty to be integrated into academic support services? 
What are they? 


Additional Costs Require Pre-Planning 


□ Explore potential funding options (e.g., reallocation of existing college resources, 
government funding, and private grants from industry partners and foundations). 

□ Acquire required funding. 

□ Set up the budgetary structure for the initiative. 


Copyright (c) 2015 by Office of Academic Affairs, City University of New York. 
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Appendix A - Postscript from the Institute of Education Sciences 
What is a practice guide? 

The Institute of Education Sciences (lES) publishes practice guides to share evidence and expert 
guidance on addressing education-related challenges not readily solved with a single program, 
policy, or practice. Each practice guide’s panel of experts develops recommendations for a 
coherent approach to a multifaceted problem. Each recommendation is explicitly connected to 
supporting evidence. Using What Works Clearinghouse (WWC) design standards, the supporting 
evidence is rated to reflect how well the research demonstrates the effectiveness of the 
recommended practices (see Table A in the Introduction for more details on levels of evidence): 

■ Strong Evidence means positive findings are demonstrated in multiple 
well-designed, well-executed studies, leaving little or no doubt that the 
positive effects are caused by the recommended practice. 

■ Moderate Evidence means well-designed studies show positive impacts, but there are 
questions about whether the findings can be generalized beyond the study samples 
or whether the studies definitively show evidence that the practice is effective. 


■ Minimal Evidence means that there is not definitive evidence that the recommended 
practice is effective in improving the outcome of interest, although there may be 
data to suggest a correlation between the practice and the outcome of interest. 


How are practice guides 
developed? 

To produce a practice guide, lES first selects 
atopic. Topic selection is informed by 
inquiries and requests to the WWC Help Desk, 
a limited literature search, and evaluation 
of the topic’s evidence base. Next, lES 
recruits a panel chair who has a national 
reputation and expertise in the topic. The 
chair, working with lES and WWC staff, 
then selects panelists to help develop the 
guide. Panelists are selected based on their 
expertise in the topic area and the belief that 
they can work together to develop relevant, 
evidence-based recommendations. Panels 
include two practitioners with expertise in the 
topic. Relevant studies are identified through 
panel recommendations and a systematic 
literature search. These studies are then 
reviewed using the WWC design standards 
by certified reviewers who rate each 
effectiveness study. The panel synthesizes 
the evidence into recommendations. 


WWC staff summarize the research and help 
draft the practice guide. lES practice guides 
are then subjected to external peer review. 
This review is done independently of the 
lES staff who supported the development 
of the guide. A critical task of the peer 
reviewers of a practice guide is to determine 
whether the evidence cited in support 
of particular recommendations is up-to- 
date and that studies of similar or better 
quality that point in a different direction 
have not been overlooked. Peer reviewers 
also evaluate whether the level of evidence 
category assigned to each recommendation 
is appropriate. After the review, a practice 
guide is revised to meet any concerns of 
the reviewers and to gain the approval of 
the standards and review staff at lES. 

Levels of evidence for What Works 
Clearinghouse practice guides 

This section provides information about 
the role of evidence in lES’s WWC practice 
guides. It describes how practice guide 
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panels determine the level of evidence 
for each recommendation and explains 
the criteria for each of the three levels of 
evidence (strong, moderate, and minimal 
evidence). The level of evidence assigned 
to each recommendation in this practice 
guide represents the panel’s judgment 
of the quality of the existing research to 
support a claim that, when these practices 
were implemented in past research, positive 
effects were observed on student outcomes. 

After careful review of the studies 
supporting each recommendation, panelists 
determine the level of evidence for each 
recommendation using the criteria in Table 
1. The panel first considers the relevance of 
individual studies to the recommendation 
and then discusses the entire evidence base, 
taking the following into consideration: 

■ Number of studies: 

■ Study designs: 

■ Internal validity of the studies: 

■ Whether the studies represent the 
range of participants and settings on 
which the recommendation is focused: 

■ Whether findings from the 
studies can be attributed to the 
recommended practice: and 

■ Whether findings in the studies 
are consistently positive. 

A rating of strong evidence refers to 
consistent evidence that the recommended 
strategies, programs, or practices improve 
student outcomes for a diverse population 
of students. In other words, there is strong 
causal and generalizable evidence. 

A rating of moderate evidence refers 
either to evidence from studies that allow 
strong causal conclusions, but cannot be 


generalized with assurance to the population 
on which a recommendation is focused 
(perhaps because the findings have not 
been widely replicated) or to evidence from 
studies that are generalizable but have 
some causal ambiguity. It also might be 
that the studies that exist do not specifically 
examine the outcomes of interest in the 
practice guide, although the studies may 
be related to the recommendation. 

A rating of minimal evidence suggests that 
the panel cannot point to a body of evidence 
that demonstrates the practice’s positive 
effect on student achievement. In some cases, 
this simply means that the recommended 
practices would be difficult to study in a 
rigorous, experimental fashion: in other 
cases, it means that researchers have not yet 
studied this practice, or that there is weak 
or conflicting evidence of effectiveness. 

A minimal evidence rating does not indicate 
that the recommendation is any less 
important than other recommendations with 
a strong or moderate evidence rating. In 
developing the levels of evidence, the panel 
considers each of the criteria in Table A. The 
level of evidence rating is determined by 
the lowest rating achieved for any individual 
criterion. Thus, for a recommendation to 
get a strong rating, the research must be 
rated as strong on each criterion. If at least 
one criterion receives a rating of moderate 
and none receives a rating of minimal, 
then the level of evidence is determined 
to be moderate. If one or more criteria 
receive a rating of minimal, then the level 
of evidence is determined to be minimal. 

The panel relied on WWC design standards 
to assess the quality of evidence supporting 
education programs and practices. The WWC 
evaluates evidence for the causal validity 
of instructional programs and practices 
according to WWC design standards. 
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Information about these standards is available 
at http://whatworks.ed.gov. Eligible studies 
that meet WWC designs standards without 
reservations or meet WWC design standards 
with reservations are indicated by bold text 
in the endnotes and references pages. 

A final note about lES practice guides: In 
policy and other arenas, expert panels 
typically try to build a consensus, 
forging statements that all its members 
endorse. Practice guides do more than 
find common ground; they create a list 
of actionable recommendations. 

Where research clearly shows which practices 
are effective, the panelists use this evidence 
to guide their recommendations. However, 
in some cases research does not provide a 
clear indication of what works. In these cases, 
the panelists’ interpretation of the existing 
(but incomplete) research plays an important 
role in guiding the recommendations. 


As a result, it is possible that two teams of 
recognized experts working independently to 
produce a practice guide on the same topic 
would come to very different conclusions. 
Those who use the guides should recognize 
that the recommendations represent, in 
effect, the advice of consultants. However, 
the advice might be better than what 
a school or district could obtain on its 
own. Practice guide authors are nationally 
recognized experts who collectively endorse 
the recommendations, justify their choices 
with supporting evidence, and face rigorous 
independent peer review of their conclusions. 
Schools and districts would likely not find 
such a comprehensive approach when 
seeking the advice of individual consultants. 

— Institute of Education Sciences 
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of topics, including the use of portfolios 
in dental education, a meta-analysis of 
the social interdependence literature, and 


investigating the mathematical education 
of future elementary teachers. Her research 
has employed diverse methods, including 
qualitative case study, survey research, 
qualitative observation and interview, meta- 
analysis, and hierarchical linear modeling. 
Dr. Young has experience as a university 
educator at Michigan State University. In 
addition, as a research analyst and certified 
reviewer for WWC, Dr. Young helps manage 
the quality control process so that all 
products meet WWC standards. She also 
manages and leads abstract and full-text 
screening of studies found in systematic 
searches, retrieves studies and other 
resources using systematic searches, and 
constructs and collaborates in constructing 
search strategies for systematic searches. 

Dr. Young received her Ph.D. in educational 
psychology and educational technology 
from Michigan State University. 
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Practice guide panels are composed of 
nationally recognized experts on the 
topics about which they are making 
recommendations. lES expects the experts 
to be involved professionally in a variety 
of matters that relate to their work as a 
panel. Panel members are asked to disclose 
these professional activities and institute 
deliberative processes that encourage critical 
examination of their views as they relate 
to the content of the practice guide. The 
potential influence of the panel members’ 
professional activities is further muted by 
the requirement that they ground their 
recommendations in evidence that is 
documented in the practice guide. In addition, 
before all practice guides are published, 
they undergo an independent external peer 
review focusing on whether the evidence 
related to the recommendations in the 
guide has been presented appropriately. 

The professional activities reported by 
each panel member that appear to be 
most closely associated with the panel 
recommendations are noted below. 

Tom Bailey, Ph.D. He is the director of the 
Community College Research Center (CCRC) 
at Teachers College, Columbia University, 
which produced many of the studies reviewed 
and used as evidence for this practice guide. 
For example, CCRC published many of the 
studies reviewed and used as evidence 
for Recommendation 4 on compressing or 
mainstreaming developmental education. 
CCRC also reported two studies used 
as evidence for Recommendation 1 
on using multiple measures to assess 
and place students into courses. 


Joanne Bashford, Ph.D. She does 
not have research reviewed for the 
recommendations presented in this practice 
guide, or any notable conflict of interest 
with the panel’s recommendations. 

Angela Boatman, Ph.D. Her research on the 
effectiveness of accelerated instructional 
approaches in developmental education 
has been reviewed for this practice guide. 

John Squires. He does not have research 
reviewed for the recommendations presented 
in this practice guide, or any notable conflict 
of interest with the panel’s recommendations. 

Michael Weiss, Ph.D. He works at MDRC, 
which produced many of the independent 
evaluations of interventions reviewed for 
this practice guide. Two of the practice 
guide’s recommendations rely in part on 
evidence from evaluations that Dr. Weiss was 
integrally involved in. Recommendation 2 
on enhanced advising and Recommendation 
6 on comprehensive, integrated, and 
long-lasting supports. Additionally, 

Dr. Weiss’ wife, Reshma Patel, played a 
substantial role in much of the evidence 
reviewed for Recommendation 3 on 
performance-based monetary incentives. 
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Appendix D. Appendix Table 1.1 


Recommendation 1. Use Multiple Measures to Assess Postsecondary Readiness and Place Students. 


Evidence Summary of Studies that Investigate the Effects of Colleges’ Use of Multiple Measures to Assess Postsecondary Readiness and 
Place Students 

Study and 
design 

Participants 

Setting 

Intervention con- 
dition as imple- 
mented in the study 

Comparison 
condition as 
implemented 
in the study 

Outcome domain and effect size 

Marwick 

(2002a)': 

Marwick 

(2002b): 

Marwick 

(2004) 

Randomized 
controlled 
trial (RCT) 

Does not 
meet WWC 
Group 

Design 

Standards 

304 students (90 were placed 
in college-level courses, and 
214 were placed in develop- 
mental level courses). 

Of the students placed in 
developmental education, 
placement condition made a 
difference for 43% of them. ^ 
From the developmental 
education sample for whom 
placement method made a 
difference, the analysis pre- 
sented here had 53 students. 

Of the students placed in 
college-level courses, place- 
ment condition made a differ- 
ence for 49% of them. 

From the college-level sample 
for whom placement method 
made a difference, the analy- 
sis presented here had 39 
students. 

One 

com- 
munity 
college 
in Illinois 

Students in the inter- 
vention condition 
were placed according 
to one of three meth- 
ods: (a) the number of 
math courses taken 
in high school,^ (b) 
“balancing” number 
of high school math 
courses and ACC- 
UPLACER score, or (c) 
student choice, but 
with advisor input and 
constrained by the 
number of high school 
math courses and 
ACCUPLACER score. 

Note that 71% of the 
304 students were 
assigned to course 
level (developmen- 
tal or college-level) 
by one of these 
three experimental 
conditions. 

Students in 
the compari- 
son condition 
were placed 
on the basis 
of a test 
score alone 
(ACCUPLACER) 

Note that 29% 
of the 304 
students were 
assigned to 
course level 
(developmen- 
tal or college- 
level), by test 
score alone. 

Develoomental education samole 

Academic achievement (passed devel- 
opmental level math): 52.4% weighted 
average completion rate for interven- 
tion groups vs. 69% completion rate in 
test score only comparison group” 

Credit accumulation (enrolled in the 
subsequent math course the next 
semester): 64.3% weighted persistence 
rate for the intervention groups vs. 
58.6% persistence rate for the test 
score only comparison group” 

Colleae-level samole 

Academic achievement (grade in col- 
lege level math): 2.5 weighted average 
course CPA in intervention groups vs. 
3.13 average course CPA in test score 
only comparison group*'* 

Academic achievement (passed col- 
lege level math): 88.9% weighted aver- 
age completion rate for intervention 
groups vs. 88.2% completion rate in 
test score only comparison group” 
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Study and 
design 

Participants 

Setting 

Intervention condition as implemented 
in the study 

Comparison 
condition as 
implemented 
in the study 

Outcome domain and 
effect size 

Ngo & Kwon 
(2014)5; 

Kwon (2014) 

Quasi-exper- 
imental 
design (QED) 

Does not 
meet WWC 
Group 

Design 

Standards 

12,854 first- 
time college 
students 

Two com- 
munity 
colleges in 
southern 
California 

In both colleges, additional points on the 
ACCUPLACER (multiple measures “boost” 
points) were awarded based on high school 
math experiences. 

College H awarded boost points based on 
self-reported high school CPA. 

College A awarded boost points based on 
highest level of math taken with a “C” or 
better, the number of years of math taken 
in high school, the length of time since 
math was last taken, and whether or not the 
student took algebra. 

Students in 
the compari- 
son condition 
were placed 
using the 
ACCUPLACER. 

Progress through devel- 
opmental education: NA 

Credit accumulation: NA® 


' This study did not meet standards because attrition exceeded WWC thresholds, and the WWC was unable to assess the comparability of the analytic sample 
at baseline. 

^This means that assignment to treatment condition placed the student in a different course level, compared to the traditional placement measure (I.e., 
ACCUPLACER score). 

^While this group was not assigned by multiple measures to courses, this intervention group is included as an Intervention with an alternate placement 
method. 

indicates a statistically significant result {p < .05). The author reported that students placed into college-level courses by both high school preparation and 
test scores received higher grades than the students placed into college-level courses by test score alone. However, college-level students placed by high 
school preparation alone performed equally as well as students placed via the placement test on the measures of academic success in the study. Among 
students placed at the developmental level, there were no differences in academic success outcomes for the multiple or alternate measures interventions, 
compared to students placed by test score alone. The author found that single-measure methods may over-select students into developmental education 
(including students who could have succeeded In college-level courses). 

^This study did not meet WWC standards because the WWC was unable to assess baseline equivalence on socioeconomic status and academic achievement. 
The authors do use baseline achievement in their estimation models, but do not have a measure of socioeconomic status. 

^This study did not present information in a way that allows standardized effect sizes to be reported. According to the authors, none of these effects were 
statistically significant, indicating that “boosted” students did as well as students placed traditionally. 
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Appendix D. Appendix Table 1.2 


Recommendation 1. Use Multiple Measures to Assess Postsecondary Readiness and Place Students. 


Correlational Studies Examining Multiple Measures for Readiness Assessment^ 

Study 

Sample (size) 

Placement Tests Examined 

Highlighted findings 

Belfield & Crosta 
(2012) 

Correlational Study 

Does not have an 
eligible design for 
WWC review 

First time com- 
munity college 
students (spe- 
cific sample 
sizes are not 
given, but 
likely are above 
18,000) 

ACCU PLACER and COMPASS 
subject specific tests (e.g., 
COMPASS algebra, pre-algebra, 
reading, writing) 

The correlation between high school (HS) CPA and grades 
in developmental courses was much stronger than the 
correlation between scores on placement tests and grades 
in developmental courses (correlations were approximately 
-I-.12* for placement test to relevant subject grade, vs. -I-.35* 
for HS CPA vs. grade).® 

Scott-Clayton (2012) 

Correlational Study 

Does not have an 
eligible design for 
WWC review 

First time com- 
munity college 
students {n = 
6,100 for math 
and 9,268 for 
English) 

COMPASS subject specific tests 
(numerical skills/pre-algebra, 
algebra, English, and a writing 
test “adapted slightly from the 
COMPASS writing module” (p. 

9) 

The correlation between the math placement test and 
grades in college math was -I-.36*, and the correlation 
between the English placement test and grades in college 
English was -1-.13*. 

The correlation between an index of high school perfor- 
mance (created from HS CPA and other variables) and 
grades in college math was -I-.45*, and the correlation 
between the high school performance index and grades in 
English was -I-.31*. 


^No studies for this recommended practice met WWC evidence standards. Due to the rigor of these correlational studies, the expert panel decided to include 
them as evidence. However, the WWC did not conduct a thorough search for studies that examine relationships between alternative measures of college 
readiness and college success. The studies discussed here were uncovered during the search for interventions that assessed the effect of using multiple or 
alternative measures. As such, these results should be considered suggestive. 

indicates a statistically significant result (p< .05) 
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Appendix D. Appendix Table 2 


Recommendation 2. Require or Incentivize Students to Attend Enhanced Advising. 


Summary of Evidence of Studies that Investigate the Effects of Enhanced Advising 

Study and 
design 

Participants 

Setting 

Intervention condition as implemented in 
the study 

Comparison con- 
dition as impie- 
mented in the study 

Outcome 
domain and 
effect size 

Cousert (1999) 

Randomized 
controlled trial 
(RCT) 

Meets WWC 
Group Design 
Standards 
Without 
Reservations 

175 students 
enrolled in 
basic skills 
reading 
courses and 
identified as 
at-risk for 
dropping 
out (based 
on the Col- 
lege Student 
Inventory) 

Ivy Tech 
State 
College 
in Central 
Indiana 

The 1 -semester New Dimensions program offered 
faculty mentoring to academically underprepared 
students, in which faculty mentors followed 
up with students’ progress on their action plan 
and referred them to other support services 
(e.g., tutoring, career counseling). Students in 
the intervention received direct faculty mentor- 
ing contact, worked to create a written student 
action plan based on the College Student Inven- 
tory, and received follow-up monitoring on the 
actions outlined in the student’s plan. Students 
were randomly selected for participation in the 
intervention, but were not required to participate 
in the mentoring program. 

The comparison group 
did not receive any 
additional mentoring 
services. All tradi- 
tional college coun- 
seling services were 
available, however, 
and thus the condi- 
tion is considered 
business-as-usual. 

Progress 
through 
developmen- 
tal education 

0 = -1-0.1 2“ 

Credit 

accumulation 
0 = -1-0.19“ 
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Appendix D. Appendix Table 2 (continued) 


Recommendation 2. Require or Incentivize Students to Attend Enhanced Advising, (continued) 


Study and 
design 

Participants 

Setting 

Intervention condition as imple- 
mented in the study 

Comparison condition as 
implemented in the study 

Outcome 
domain and 
effect size 

Scrivener & 
Weiss (2009): 
Scrivener & 

Pih (2007): 
Scrivener & Au 
(2007) 

RCT 

Meets WWC 
Group Design 
Standards 
Without 
Reservations 

2,139 students 
who had experi- 
enced academic 
difficulties, 
operationalized 
as not pass- 
ing courses, or 
withdrawing 
from courses. 
65% of the full 
sample enrolled 
in developmen- 
tal coursework 
in the first 
year.® 

Two com- 
munity 
colleges 
in Ohio 

The Opening Doors program provided 
college counseling to students in a more 
“intensive, comprehensive, and personal- 
ized manner relative to traditional college 
standards” (p. ES-3). Students enrolled in 
the intervention received personalized 
counseling services on a range of topics, 
including course selection, registration, 
financial aid, tutoring, work-based learn- 
ing efforts, juggling work and school, 
career aspirations, and personal issues. 
Intervention students were eligible to 
receive a small stipend per semester of 
$150, for a two-semester period. 

The comparison group did 
not receive any additional 
benefits beyond those usu- 
ally associated with commu- 
nity college enrollment. As 
such, all comparison group 
members were eligible to 
meet with a college coun- 
selor, but scheduling and 
activities were not assigned 
or regimented. The compari- 
son group students did not 
receive a stipend. 

Enrollment: 
g = -1-0.07“ 

Academic 

achievement: 

0 = -O.Ol“ 

Credit accu- 
mulation: 

0 = +O.O9*'° 

Degree 

attainment: 

0 = -O.28“ 


® MDRC defined “academic difficulty” as earning no more than 75% of credits attempted. Other sample criteria included students being aged 18-34 years 
old; having family income below 250% of poverty level; being beginning or continuing students who completed fewer than 13 credits and had experienced 
academic difficulties (indicated by not passing courses or withdrawing from courses), having a high school diploma or GED, and did not have an associate’s 
degree. 

indicates a statistically significant result {p< .05) 
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Appendix D. Appendix Table 2 (continued) 


Recommendation 2. Require or Incentivize Students to Attend Enhanced Advising, (continued) 


Study 

and 

design 

Participants 

Setting 

Intervention condition as implemented in the study 

Comparison 
condition 
as impie- 
mented in 
the study 

Outcome 
domain 
and effect 
size 

Visher et 
al. (2010) 

RCT 

Meets 

WWC 

Croup 

Design 

Stan- 

dards 

Without 

Reserva- 

tions 

1,249 
students 
enrolled 
in a devel- 
opmental, 
lower-level 
math course 
required for 
advance- 
ment toward 
a degree or 
certificate. 

One 

com- 
munity 
col- 
lege in 
Texas 

The Beacon Mentoring Program offered support for accessing 
enhanced advising in the context of students’ required mathematics 
classes during students’ second semester (i.e.. Spring 2008). Program 
goals were to increase students’ use of tutoring and advising in order 
to improve pass rates in high-failure math courses, credits earned, 
and semester-to-semester persistence. Mentors were college employ- 
ees who worked on campus and had previously participated in a 
training program to be mentors. In the intervention group classes, 
the mentors were required to arrange with math instructors to meet 
with their classes at least three times during the semester for 5-10 
minutes at the beginning of class. The mentor provided information 
on campus resources and notified students of important dates. The 
mentor sometimes went with the class to the student success center, 
the financial aid office, or other useful areas on campus. The men- 
tors shared their email address, office location, and office number 
with students and encouraged them to contact or visit them if they 
needed anything. The mentors reminded students about registration 
toward the end of the semester. 

Comparison 
group stu- 
dents did not 
receive men- 
toring ser- 
vice. As such, 
the students 
in this condi- 
tion could be 
considered 
business- 
as-usual 
because they 
had access 
to traditional 
college 
counselors. 

Progress 
through 
develop- 
mental 
education: 
g = -1-0.07“ 

Academic 

achieve- 

ment: 

g = -1-0.06“ 

Credit accu- 
mulation: 
g = +0A5* 
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Appendix D. Appendix Table 3.1 


Recommendation 3. Offer Students Performance-Based Monetary Incentives. 


Evidence Summary of Studies that Investigate Effects of Performance-Based Monetary Incentives Alone or Minimal Additional Support 

Study and 
design 

Participants 

Setting 

Intervention condition as 
implemented in the study 

Comparison condition as 
impiemented in the study 

Outcome 
domain and 
effect size 

Brock & Rich- 
burg-Hayes 
(2006); Rich- 
burg-Hayes 
et al. (2009): 
Barrow et al 
(2012) 

Randomized 
controlled trial 
(RCT) 

Meets WWC 
Group Design 
Standards 
Without 
Reservations 

1,019 low- 
income parents 
ages 18-34. 

64% of the 
participants 
enrolled in 
at least one 
developmen- 
tal education 
course within 
the first 2 
years. 

Two New 
Orleans, 
Louisiana- 
based 

community 

colleges: 

Delgado 

Community 

College and 

Louisiana 

Technical 

College- 

West 

Jefferson 

In addition to any other financial 
aid for which students quali- 
fied for, the Louisiana Opening 
Doors program offered eligible 
low-income parents a $1,000 
monetary award for each of 
two semesters ($2,000 total) for 
maintaining at least half-time 
enrollment and a 2.0 CPA. Stu- 
dents in the intervention group 
also received advising and moni- 
toring support (i.e., dedicated 
program advisors, who helped 
verify enrollment and monitored 
students). The program required 
that intervention students meet 
with a counselor periodically. 

The comparison group was 
considered “business as usual.” 
Participants in the comparison 
group received the standard 
financial aid supports, includ- 
ing Pell Grants. They could 
access academic advising and 
counseling which was avail- 
able to all students. They did 
not receive an Opening Doors 
scholarship and did not have 
counselors who monitored 
their academic performance. 

Enrollment 

0 = -1-0.14*" 

Credit accumula- 
tion g = -1-0.20“ 

Academic achieve- 
ment ^ = -1-0.17“ 


" * indicates a statistically significant result {p< .05) 
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Appendix D. Appendix Table 3.1 (continued) 


Recommendation 3. Offer Students Performance-Based Monetary Incentives. 


Study and design 

Participants 

Setting 

Intervention condition as 
implemented in the study 

Comparison condition 
as implemented in the 
study 

Outcome 
domain and 
effect size 

Cha& Patel (2010); 
Mayer, Patel, & Guti- 
errez (2015) 

RCT 

Meets WWC Croup 
Design Standards 
Without Reservations 

2,285 students, 
required to (a) 
have an expected 
family contribu- 
tion to college 
expenses of $0 and 
(b) be the parent 
of a minor child. 
Approximately 40% 
of the study sample 
enrolled in at least 
one developmental 
education course, 
during the study 
period. 

Three 
community 
colleges in 
Ohio 

Students were eligible for 
awards up to $1,800 per 
academic year for earning 
a C or better in 12 or more 
credits per term, or up 
to $900 for meeting that 
benchmark in 6-11 credits 
per term. 

Comparison group stu- 
dents were eligible for 
services normally avail- 
able to students. 

Enrollment 

0 = -1-0.06”" 

Credit accumula- 
tion g = -1-0.07”" 

Degree attain- 
ment g = -1-0.06"" 

Patel & Rudd (2012); 
Richburg-Hayes et al. 
(2011) 

RCT 

Meets WWC Croup 
Design Standards 
Without Reservations 

1,502 students 
between the ages 
of 22 and 35. Stu- 
dents were required 
to be living on their 
own, enrolled in at 
least 6 credit hours 
at intake, eligible 
for a federal Pell 
Grant, and required 
to take at least one 
remedial course. 

Two com- 
munity 
colleges in 
New York 
City 

Need-based scholarships 
were paid directly to indi- 
vidual students conditionally 
upon meeting certain aca- 
demic benchmarks; students 
could receive up to $2,600 
(for 2 semesters) or $3,900 
(for 2 semesters and a sum- 
mer term). 

Comparison group stu- 
dents were eligible for 
services normally avail- 
able to students. 

Enrollment 
g = -1-0.07”" 

Credit accumula- 
tion g = -1-0.06”" 

Academic achieve- 
ment g = -1-0.01"" 

Degree attain- 
ment g = -1-0.05”" 
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Recommendation 3. Offer Students Performance-Based Monetary Incentives. 


Evidence Summary of Studies that Investigate the Effects of Performance-Based Monetary Incentives Plus Other Supports 

Study and 
design 

Participants 

Setting 

Intervention condition as implemented in 
the study 

Comparison condition 
as impiemented in the 
study 

Outcome 
domain and 
effect size 

Miller et al. 
(2011); Binder 
et al. (2015) 

RCT 

Meets WWC 
Croup Design 
Standards 
Without 
Reservations 

1,081 low 
income 
students 
(approxi- 
mately 56% in 
developmen- 
tal education 
courses) 

One 

university 
in New 
Mexico 

Conditional on earning a maintaining full time 
enrollment and a C average, students were eligible 
for $1,000 in financial aid per semester, for four 
semesters (this award is in addition to any other 
financial aid students receive). Students were also 
offered enhanced advising services. 

In the VISTA program, students in the intervention 
group were assigned an academic advisor to work 
with throughout the term, and were required to 
see their advisor at least twice. Students in the 
VISTA program saw their counselor an average of 
five times during their first semester. A student 
in the intervention group was more likely than a 
student in the comparison group to have holistic 
advising because he or she saw the same advisor 
multiple times during the academic term, rather 
than different advisors each time they came to 
advising (Binder et al., 2015). 

Comparison group stu- 
dents were eligible for 
services normally avail- 
able to students. Students 
in the comparison group 
received any available 
counselor when they went 
for advising, and were 
only required to see a 
counselor once. Compari- 
son group students saw 
an advisor about three 
times their first semester 
(Binder et al., 2015). 

Enrollment 
g = -0.09“ 

Credit 

accumulation 

0 = 0.00“ 

Academic 

achievement 

0 = +O.18*'2 

Degree 

attainment 

0 = -1-0.10“ 


indicates a statistically significant result {p< .05) 
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Appendix D. Appendix Table 3.2 (continued) 

Recommendation 3. Offer Students Performance-Based Monetary Incentives. 


Study and design 

Participants 

Setting 

Intervention condition as imple- 
mented in the study 

Comparison condition 
as impiemented in the 
study 

Outcome 
domain and 
effect size 

Patel & Valenzuela 
(2013) 

RCT 

Meets WWC 

Croup Design 
Standards Without 
Reservations 

1,028 low 
income, 

Latino male 
students (58% 
of the sample 
enrolled in 
developmen- 
tal education 
courses by 
the end of 
first year) 

Pima Com- 
munity 
College in 
Tucson, AZ 

The Adelante program is a 3-semester 
program, whereby all students receiv- 
ing the intervention were eligible for 
performance-based awards up to $1,500 
per semester. Students received $150 
for attending an orientation, and $150 
for meeting with an Adelante advisor 
twice during the semester. The remaining 
$1200 payment was based on academic 
performance and participation in aca- 
demic tutoring, workshops, and platicas 
sessions (in which small groups of Latino 
men discuss challenges and issues they 
face). 

The Adelante program provided des- 
ignated advisors to students in the 
intervention group for all three program 
semesters. Adelante program students 
were required to attend a minimum of 
two semi-structured advising sessions, 
focused on the development and follow- 
through of a personal education plan. 

The comparison group 
did not receive an 

Adelante program ori- 
entation. They had the 
opportunity to have 
general advising offered 
at the college, but were 
not incentivized to do so. 
They could participate in 
tutoring and academic 
workshops, but had no 
scholarship incentive to 
do so. In the comparison 
group, general advising 
was provided as needed, 
as triaged through the 
college’s online portal 
system. 

Enrollment 
g = -h0.08“ 

Credit 

accumulation 

0 = -1-0.10“ 

Academic 

achievement 

0 = -tO.O3“ 

Degree 
attainment 
g = -i-.04“ 
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Appendix D. Appendix Table 3.2 (continued) 


Recommendation 3. Offer Students Performance-Based Monetary Incentives. 


Study and design 

Participants 

Setting 

Intervention condition as 
implemented in the study 

Comparison condition 
as implemented in the 
study 

Outcome 
domain and 
effect size 

Sommo et al. (2014) 

RCT 

Meets WWC Croup 
Design Standards 
Without Reservations 

1,075 low- 
income col- 
lege students 
placed into 
developmen- 
tal math 

Multiple 
campuses 
of Hills- 
borough 
Community 
College in 
Tampa, FL 

The Mathematics Access Per- 
formance Scholarship program 
provided performance-based 
monetary incentives of up to 
$1,800 for successfully completing 
a three-course sequence within 
three consecutive semesters plus a 
summer term. Low-income stu- 
dents in developmental math were 
eligible and received monetary 
awards for participation in an 
on-campus MathLab and passing 
grades in the course sequence 
through college-level math. 

The comparison group 
received standard col- 
lege services. The only 
thing that deviated from 
“business-as-usual” is 
that study participants 
(whether in the treat- 
ment or comparison 
groups) were allowed 
access to reserved 
spaces in the math- 
ematics courses that 
were not available to 
non-participants. 

Enrollment 
g = +0.24* 

Progress through 
developmental 
education 
g = - 1 - 0 . 18 * 

Credit 

accumulation 

0 = -1-0.01“ 
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Appendix D. Appendix Table 4 


Recommendation 4. Compress or Mainstream Developmental Education with Course Redesign. 


Evidence Summary of Studies that Investigate the Effects of Mainstreaming, Compression, and Course Redesign in Developmental 
Education 

Study and 
design 

Participants 

Setting 

Intervention condition as 
implemented in the study 

Comparison condition as imple- 
mented in the study 

Outcome 
domain and 
effect size 

Hodara & 

Jaggars 

(2014) 

Quasi- 

Experimental 
Design (QED) 

Meets WWC 
Group 

Design Stan- 
dards With 
Reservations 

7,148 stu- 
dents placed 
into devel- 
opmental 
education 

courses 

Three 

com- 
munity 
colleges 
in New 

York City 

The intervention was a short- 
ened developmental writing 
sequence (6-7 credit hours), 
designed to prepare students for 
two college-level English com- 
position courses, required of all 
degree programs. The shorter, 
accelerated sequence focused on 
students’ writing in response to 
readings and class discussions. 
The curriculum did not empha- 
size a review of grammar or 
usage rules, or appear to teach 
writing through discrete skills 
instruction. 

The comparison was the usual devel- 
opmental writing sequence offered 
in these colleges (8-12 credit hours), 
designed to prepare students for 
two college-level English composi- 
tion courses, required of all degree 
programs. In the traditional, longer 
sequence, the lower-level course 
emphasized grammar instruction 
and paragraph development, and the 
next level of the course sequence 
offered a review of grammar and 
emphasis on paragraph and essay 
writing. 

Progress through 

developmental 

education: 

0 = +O.22*'3 

Credit accumula- 
tion: g = -1-0.13* 

Degree attain- 
ment: g = - 1 - 0 . 16 * 


* indicates a statistically significant result (p < .05) 
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Appendix D. Appendix Table 4 (continued) 


Recommendation 4. Compress or Mainstream Developmental Education with Course Redesign. 


Study and 
design 

Participants 

Setting 

Intervention condition as 
implemented in the study 

Comparison condition as 
impiemented in the study 

Outcome domain 
and effect size 

Cho et al. 
(2012)'": 
Jenkins et al. 
(2010) 

QED 

Does not 
meet WWC 
Croup 

Design 

Standards 

1,184 stu- 
dents placed 
into devel- 
opmental 
writing 

One com- 
munity 
college in 
the mid- 
Atlantic 

Community College of Baltimore 
County’s Accelerated Learning 
Program (ALP) mainstreamed 
developmental-level students into 
college-level English composi- 
tion. The students also received 
supplemental instruction for an 
additional hour per course meet- 
ing. The college-level English and 
supplemental instruction course 
are taught by the same instruc- 
tor with integrated syllabi and 
activities. 

The ALP comparison group was 
composed of students who were 
only enrolled in the traditional, 
highest level developmental Eng- 
lish course (ENCL 052). 

ALP students out- 
performed non-ALP 
students on college- 
level coursework 
completion rates and 
persistence to the 
next year* (nr). There 
was not a differ- 
ence on college-level 
course grades or on 
degree attainment/ 
transfer (nr).'^ 

Edgecombe 
et al. (2014)'s 

QED 

Does not 
meet WWC 
Croup 

Design 

Standards 

3,529 stu- 
dents in 
developmen- 
tal education 

One com- 
munity 
college in 
California 

A one-semester accelerated 
course in English was offered as 
a preparatory course in a pilot 
learning community, and this 
group formed the pool for the 
intervention group in the study. 
Later, the accelerated option was 
offered to all students on campus. 

The 2-semester traditional pathway 
consisted of three developmental 
courses, which was considered 
“business as usual.” Students who 
enrolled in this course sequence 
and met propensity-score match- 
ing criteria with the intervention 
group participants formed the pool 
for the comparison group. 

Over a 5-year follow- 
up period, accelerated 
course participants 
were more likely to 
complete college-level 
English* (nr), earn 
more college course 
credits* (nr), and 
more likely to earn a 
degree,* (nr)."' 


'''This study did not meet WWC standards because the authors used imputation for some covariates and outcomes; the WWC currently does not allow imputation 
for covariates. 

nr indicates not reported. This study did not present information in a way that allows standardized effect sizes to be reported. See Cho et al. (2012), Table 3, p. 
10 . 

This study did not meet WWC standards because an acceptable pre-intervention measure of academic achievement was not available. The authors did control 
for measures of college achievement and student socioeconomic status. 

'^nr indicates not reported. This study did not present information in a way that allows standardized effect sizes to be reported. See Edgecombe et al. (2014), 
Table A. 2, p. 30. 
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Appendix D. Appendix Table 4 (continued) 

Recommendation 4. Compress or Mainstream Developmental Education with Course Redesign. 


Study and 
design 

Participants 

Setting 

Intervention condition as 
implemented in the study 

Comparison condition as 
implemented in the study 

Outcome domain 
and effect size 

Zeiden- 
berg et 
al. (2010): 
Jenkins et 
al. (2009)’« 

QED 

Does not 
meet WWC 
Group 

Design 

Standards 

1,564 stu- 
dents in adult 
basic edu- 
cation and 
workforce 
development 
programs 
(i.e., enrolled 
in at least 

one career 

and technical 

education 

course) 

14 com- 
munity 
colleges in 
Washington 

l-BEST is a model in which “a basic 
skills instructor and an occupa- 
tional instructor team teach occu- 
pational courses with integrated 
basic skills content, and students 
receive college-level credit for the 
occupational coursework" (Zeiden- 
berg et al., 2010, p. 1). 

Students in the comparison 
condition were in workforce 
classes but did not participate 
in l-BEST. 

Credit Accumulation: 
(25 vs. 17 credits 
over 3 years)* '^'2° 

Degree attainment: 
g = +].U* 


'®These studies did not meet WWC standards because they did not use an acceptable pre-intervention measure of academic achievement in the estimation 
models. 

indicates a statistically significant resuit (p< .05) 

^“This study did not present enough information for the computation of a standardized effect size for this outcome. See Zeidenberg et al. (2010), Table 6, p. 
14. 
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Recommendation 5. Teach Students How to Be Self-Regulated Learners. 


Evidence Summary of Studies that Investigate the Effects of Interventions to Teach Developmental Education Students How to Be Self- 
Regulated Learners 

Study and design 

Participants 

Setting 

Intervention condition as imple- 
mented in the study 

Comparison 
condition as 
implemented in 
the study 

Outcome 
domain 
and effect 
size 

Zimmerman et al. (2011); Hudes- 
man, Crosby, Flugman, Issac, 
Everson, & Clay (2013) 

Randomized controlled trial 
(RCT) 

Meets WWC Group Design Stan- 
dards Without Reservations 

140 students 
enrolled in 
developmen- 
tal mathemat- 
ics courses. 

One 

compre- 
hensive 
college in 
the United 
States 

A semester-long classroom interven- 
tion designed to infuse instruction in 
self-regulated learning into the math 
course. 

The comparison 
condition received 
conventional 
instruction in their 
respective devel- 
opmental educa- 
tion courses. 

Progress 
through 
develop- 
mental 
education 
g = -1-0.49*21 

Academic 
achievement 
0 = -1-0.54* 


* indicates a statistically significant result {p< .05) 
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Appendix D. Appendix Table 6 


Recommendation 6. Implement Comprehensive, Integrated, and Long-Lasting Support Programs. 


Evidence Summary of Studies that Investigate the Effects of Comprehensive, Integrated, and Long-Lasting Support Programs with 
Students Placed in Developmental Education 

Study and 
design 

Participants 

Setting 

Intervention condition as 
implemented in the study 

Comparison con- 
dition as impie- 
mented in the 
study 

Outcome domain 
and effect size 

Scrivener et al. 
(2015); Scrivener 
& Weiss (2013): 
Scrivener, Weiss & 
Sommo (2012) 

Randomized con- 
trolled trial (RCT) 

Meets WWC 

Group Design 
Standards Without 
Reservations 

896 low income stu- 
dents (based on family 
income below 200% of 
poverty level, or Pell 
grant eligibility), who 
needed one or two 
developmental educa- 
tion courses. 

The study sample 
inclusion criteria also 
required that students 
be: New York City resi- 
dents, a new student 
or continuing student 
with 12 or fewer cred- 
its, willing to attend 
college full time, and 
enrolled in an ASAP- 
eligible major. 

Three 

com- 
munity 
colleges 
in New 

York City 

The ASAP program includes the 
following components: required 
full-time enrollment; consoli- 
dated block scheduling in the 
first year; a noncredit seminar 
covering topics such as goal- 
setting and academic planning; 
comprehensive student advis- 
ing services, tutoring services, 
career and employment ser- 
vices; a tuition waiver; free 
public transportation vouchers; 
and free textbooks for classes. 

Control group 
members received 
the college’s stan- 
dard services and 
courses. Although 
details are not pro- 
vided about these 
standard services, it 
is stated that these 
standard college 
services are far less 
intensive than those 
offered by the ASAP 
program. 

Postsecondary 

enrollment: 

0 = 4-0.12"^ 

Progress through 

developmental 

education: 

0 = 4-0.50*22 

Credit accumula- 
tion: g = 4-0.28* 

Academic achieve- 
ment: g = 4-0.18* 

Upward transfer: 

0 = +O.29* 

Degree attainment: 

0 = 4-0.53* 


* indicates a statistically significant result {p < .05) 
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Please note that there will still be some footnotes in the guide. Studies used as evidence in support 
of a recommendation are indicated by bold text in the endnotes and references pages. 


1. Among all first-year undergraduates in the United States, about one-third enrolled in at least 
one developmental course, and among community college students, this proportion is higher 
(approximately 40 percent) (National Center for Education Statistics, 2013). See also: Bailey, 2009; 
Bettinger, Boatman, & Long, 2010; Bettinger& Long, 2009; Long & Boatman, 2013; Zachry Rutschow 
& Schneider, 2011. 

2. Attewell, Lavin, Domina, & Levy, 2006; Table 2, p. 904. See also Radford et al., 2012. 

3. Zachry Rutschow & Schneider, 2011. 

4. Remedial education was a term initially used to describe the function that junior and community 
colleges performed by offering courses that were not at a college level. Students and faculty who 
taught pre-college level courses found “remedial education” to be stigmatizing. “Developmental 
education” became the more favored term to emphasize how faculty work to developmentally 
and incrementally help “underprepared college students attain their academic goals,” (Boylan, 
2002, pg. 3). As the U.S. college completion agenda and college readiness standards have gained 
prominence, “college ready” is a term used to keep the focus on students succeeding in college 
and completing their degree. See also: Bailey, Jaggars, & Scott- Cl ay ton, 2013. 

5. National Center for Education Statistics, 2013. 

6. Howell, Kurlaender, & Grodsky, 2010; Center for Education Policy Research, 2015. 

7. Barnett, Fay, Bork, & Weiss, 2013; Tierney & Garcia, 2011. 

8. Scott-Clayton, 2012; Belfield & Crosta, 2012. 

9. Scrivener et al., 2015. 

10. It is typical in WWC reviews of research for only a relatively small proportion of the eligible studies 
to be rated as meeting standards (overall, about 23% of eligible studies that the WWC has reviewed 
meet standards at some level). Consistent with previous WWC experiences, the most likely reason 
a study did not meet standards is that it had issues with group comparability at baseline (e.g., the 
groups were too far apart on important pretest measures; see the WWC Procedures and Standards 
Handbook. Version 3.0) . 

1 1 . This includes randomized control trials (RCT) and quasi-experimental design studies (QED). Studies 
not contributing to levels of evidence include single-case designs (SCD) evaluated with WWC pilot 
SCD standards and regression discontinuity designs (RDD) evaluated with pilot RDD standards. 

12. The research may include studies generally meeting WWC design standards and supporting the 
effectiveness of a program, practice, or approach with small sample sizes or other conditions of 
implementation or analysis that limit generalizability. The research may include studies that support 
the generality of a relation but do not meet WWC design standards; but, they have no major flaws 
related to internal validity other than lack of demonstrated equivalence at pretest for QEDs. QEDs 
without equivalence must include a pretest covariate as a statistical control for selection bias. 
These studies must be accompanied by at least one relevant study meeting WWC design standards. 

13. American Educational Research Association, American Psychological Association, and National 
Council on Measurement in Education (1999). 

14. E.g., Scott-Clayton, 2012. 

15. Complete College America, 2012. 

16. According to Ngo, Kwon, Melguizo, Bos, and Prather (2013), a lawsuit filed by the Mexican-American 
Legal Defense and Educational Fund provided the impetus for California’s policy change. In 1991, 
the Fund challenged the inequity of the Matriculation Act of 1986, which mandated placement 
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testing as a part of enrollment and matriculation services in community colleges. Ngo et al. (2013) 
reported: “The lawsuit claimed that the California Community College Chancellor’s Office failed 
to monitor appropriate use of placement tests, resulting in large proportions of Latino students 
being placed in remediation. The lawsuit was settled outside of court, but Title 5 of the California 
Code of Regulations was revised in 1992 to mandate the use of multiple measures in placement 
decisions. The goal was to reduce the “disproportionate impact” of placement tests on different 
racial and ethnic groups” (p. 2, footnote). 

17. E.g., Westrick & Allen, 2014. 

18. Marwick, 2002a; Ngo & Kwon, 2014. 

19. Marwick, 2002a. 

20. Ngo & Kwon, 2014. 

21. Belfield & Crosta, 2012; Scott-Clayton, 2012. 

22. See “Using multiple measures” on the Bill and Melinda Gates Foundation website: www. 
gatesfoundation.org; Bracco et al., 2014; Burdman, 2012; Research for Action, 2015; Two-Year 
College English Association, 2014. 

23. See: http://rpqroup.ora/proiects/multiDle-measures-assessment-proiect . The RP Group is a 
membership organization of individual California community colleges and districts, focused on 
providing research, evaluation, professional development, and technical assistance to support 
evidence-based decision-making. Individuals, colleges, and organizations not in California’s 
community college system may join as associate members. See also the Education Commission of 
the States’ (ECS) national scan of states’ implementation of multiple measures for course placement: 
http://www.ecs.ora/movina-from-sinqle-to-multiple-measures-for-colleqe-course-placement. ECS 

also refers to resources from Research for Action and WestEd, also cited in this guide. 

24. Hodara, Jaggars, & Karp, 2012. 

25. Duffy, Schott, Beaver, & Park, 2014. 

26. “LBCC Promise Pathways,” 2014. 

27. Willett, 2013. Willett (2013) investigated the relative effectiveness of various placement indicators 
on two main outcomes: 1) the students’ placement level of first English course, and first math 
course attempted; and 2) receiving a passing grade in the students’ first attempted English and 
math courses (English and math analyzed separately). The predictors that were used in regression 
models were: a) the most recent California Standards test in English or math; b) count of “A-C” or 
college preparatory courses completed (a set of seven categories of high school courses to be 
eligible for admission to the University of California or California State University); c) level of most 
recent high school course in English or math; d) grade point in most recent high school course 
in English or math; e) high school grade point average excluding English (or math); and f) level of 
first college course attempted in English (or math), used only for regressions predicting passing 
grades in those courses). 

28. Willett et al., 2015. 

29. Melguizo et al., 2014. 

30. Adapted from: Cal-PASS Plus, & RP Croup, (n.d). Multiple measures assessment project - Pilot college 
- Developing a research plan. Retrieved from the RP Croup web site: http://rpgroup.org/system/ 
files/MMAP-Sample%20Research%20Plan_10.31.15.pdf and Cal-PASS Plus, & RP Croup, (n.d.). Multiple 
measures assessment project - Data match guide to Cal-PASS Plus. Retrieved from the RP Group 
web site: http://rpgroup.org/system/files/DataMatch%20Guide_l 0.26.1 5.pdf 

31. Grubb, 2001. 

32. Scrivener & Weiss, 2009. 

33. Escobedo, 2007. 

34. Jackson, 2014. 

35. Robbins, 2013; Education Advisory Board, Complete College America, 2012. 
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36. Kalamkarian & Karp, 2015. 

37. Scrivener & Weiss, 2009; Visher et al., 2010. 

38. Cousert, 1999; Visher et al., 2010. 

39. Scrivener & Weiss, 2009. 

40. Cousert, 1999. 

41. Ford, 2007; McClellan, 2007. 

42. Self, 2008; Kalamkarian & Karp, 2015. 

43. Kalamkarian & Karp, 2015. 

44. Scrivener & Weiss, 2009. 

45. Visher et al., 2010. 

46. Scrivener & Weiss, 2009. 

47. Scrivener & Weiss, 2009. 

48. Lynch-Holmes, Troy, & Ramos, 2012. 

49. Sample enhanced advising planning tool from Ivy Tech State College (p. 118) from: Cousert, D. 
(1999). The effects of a mentoring intervention program on retention of students in a community 
college (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI 
No. 304550777). This study meets WWC standards without reservations. 

50. Visher, Butcher, & Cerna, 2010, p. 13. 

51. Scrivener & Weiss, 2009. 

52. Scrivener & Weiss, 2009, p. 27. 

53. Visher et al., 2010. 

54. Patel, Richburg-Hayes, de la Campa, & Rudd, 2013; Mayer, Patel, Rudd, Ratledge, & Blake, 2015. 

55. Brock & Richburg-Hayes, 2006; Cha & Patel, 2010; Patel & Rudd, 2012. The effectiveness of 
performance-based monetary incentives alone were evaluated in two Ohio colleges in Cha & Patel 
(2010) and in two New York colleges in Patel & Rudd (2012). Note that in the Louisiana Opening 
Doors program (Brock & Richburg-Hayes, 2006), counseling was primarily focused on monitoring 
students’ compliance with program requirements and not on providing academic advising or 
supports to work through personal challenges to their academic progress. The Louisiana Opening 
Doors intervention differed from the coupling of enhanced advising, with performance-based 
monetary incentives, in the VISTA program studied by Miller et al. (201 1) and the Adelante program 
studied by Patel & Valenzuela (2013). 

56. Brock & Richburg-Hayes, 2006; Patel & Rudd, 2012. 

57. Cha & Patel, 2010. 

58. Brock & Richburg-Hayes, 2006; Patel & Rudd, 2012. 

59. Cha & Patel, 2010; Patel & Rudd, 2012. 

60. Miller, Binder, Harris, & Krause, 2011; Patel & Valenzuela, 2013; Sommo et al., 2014. 

61. Miller et al., 2011; Patel & Valenzuela, 2013. 

62. Patel & Valenzuela, 2013. 

63. Welbeck, Ware, Cerna, & Valenzuela, 2014. 

64. Adapted from source: Howto Budget a Performance-Based Scholarship, Box 2 (p. 15) from Welbeck, 
R., Ware, M., Cerna, O., & Valenzuela, I. (2014, February). Paying it forward: A technicai assistance 
guide for deveioping and impiementing performance-based schoiarships. New York, NY: MDRC. 

65. Mayer et al., 2015b. 

66. Patel & Rudd, 2012. 

67. Welbeck et al., 2014. 

68. Sommo et al., 2014. 

69. Adapted from: Welbeck, R., Ware, M., Cerna, O., & Valenzuela, I. (2014). Paying it forward: A 
technicai assistance guide for deveioping and impiementing performance-based schoiarships. New 
York, NY: MDRC. 
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70. Jaggars, Hodara, Cho, &Xu, 2015; Kosiewicz, Ngo, & Fong, 2016; Bracco, Austin, Bugler, & Finkelstein, 
2015. 

71. Edgecombe, 2011, p. 4. 

72. Zachry Rutschow & Schneider, 2011. 

73. Hodara & Jaggars, 2014. 

74. Edgecombe, Jaggars, Xu, & Barragan, 2014. 

75. Edgecombe, Jaggars, Baker, & Bailey, 2013b; This study does not meet WWC evidence standards 
because the preintervention equivalence of the intervention and comparison groups could not be 
assessed. Additionally, footnote 12 indicated that 16% of students were both the intervention and 
comparison groups, which is a confounding factor in the analysis. 

76. Fong & Visher, 2013; This study does not meet WWC evidence standards because the pre- 
intervention equivalence of the intervention and comparison groups could not be assessed. 

77. Zachry Rutschow & Schneider, 2011. 

78. Cho, Kopko, Jenkins, and Jaggars (2012) evaluated the effectiveness of the Accelerated Learning 
Program (ALP), however, the study did not meet WWC standards because the authors used 
imputation for some covariates and outcomes; the WWC currently does not allow imputation for 
covariates. 

79. Zachry Rutschow & Diamond, 2015; This study does not meet WWC evidence standards because 
the intervention and comparison groups could not be established as equivalent at baseline on both 
a continuous measure of prior academic achievement and a measure of students’ socioeconomic 
status. 

80. Zeidenberg, Cho, & Jenkins, 2010; Jenkins, Zeidenberg, & KienzI, 2009. 

81. Yamada, 2014; This study does not meet WWC evidence standards because the intervention and 
comparison groups could not be established as equivalent at baseline, due to the use of a binary 
measure of prior academic achievement. Additionally, the comparison group was drawn from a 
prior cohort, rather than comparing two groups from the same time period, which is a confounding 
factor in the analysis. 

82. Hodara & Jaggars, 2014. 

83. Hodara and Jaggars (2014) also investigated the effects of an accelerated math course compared 
to the traditional length developmental math course, the comparison group for this analysis had 
only one college in contrast to five colleges in the intervention group. In this situation, the effects 
of the intervention cannot solely be attributed to accelerated math instruction, because the college 
in the comparison group may have unique characteristics affecting the results. 

84. See also Jenkins, Zeidenberg, & KienzI, 2009. 

85. Adapted from: Nodine, T., Dadgar, M., Venezia, A., & Bracco, K. R. (2013) Acceleration in 
developmental education. San Francisco, CA: WestEd. Retrieved from http://www.wested.org/ 
online_pubs/resourcel281.pdf. 

86. Adapted from: Nodine, T., Dadgar, M., Venezia, A., & Bracco, K. R. (2013) Acceleration in 
developmental education. San Francisco, CA: WestEd. Retrieved from http://www.wested.org/ 
online_pubs/resourcel281.pdf. 

87. Jaggars et al., 2015. 

88. Jaggars et al., 2015. 

89. Fay, Bickerstaff, & Hodara, 2013. 

90. Zachry Rutschow & Diamond, 2015; Katiraie, 2007. 

91. Boylan, 2002; Shults, 2000. 

92. Bain, 2004; Coalition of Academic Workforce, 2012; Creen, 2007; Linder, 2012. 

93. Edgecombe, Cormier, Bickerstaff, & Barragan, 2013a; Bracco, Austin, Bugler, & Finkelstein, 2015; 
Kosiewicz, Ngo, & Fong, 2016. 

94. Adapted from: Nodine, T., Dadgar, M., Venezia, A., & Bracco, K. R. (2013) Acceleration in 
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developmental education. San Francisco, CA: WestEd. Retrieved from http://www.wested.org/ 
online_pubs/resourcel281.pdf. 

95. This is called self-regulated learning in the research literature; it has been well-researched among 
K-12 students. Many scholars consider self-regulated learning to be one aspect of metacognition 
and have discussed the interrelationships of metacognition, self-regulation, and self-regulated 
learning (e.g., Al-Harthy, Was, & Issacson, 2010; Dinsmore, Alexander, & Loughlin, 2008; Kaplan, 
2008; Sperling, Howard, Staley, & DuBois, 2004). See also Courgey (1998) for a discussion of 
teaching metacognition and self-regulated learning in basic skills instruction. 

96. See for example, Zimmerman, 2002. 

97. Zimmerman, Moylan, Hudesman, White, & Flugman, 2011. 

98. Zimmerman et al., 2011. 

99. Weinstein, Meyer, Husman, Van Mater Stone, & McKeachie, 2006. 

100. Zimmerman et al., 2011. 

101. Weinstein et al., 2006, p. 304. 

102. Weinstein et al., 2006. 

103. Brophy, 2004. 

104. Hudesman et al., 2013b, p. 17. 

105. Hudesman et al., 2014. 

106. In Hudesman et al. (2014), the log form contained the following questions: What was your 
goal for your students and/or yourself this period? What were the mathematics content and 
metacognitive strategies your students and/or you were planning to use to achieve this goal? 
What actually happened? How will the results of this cycle inform your instruction in the future? 
What worked that you will continue? What adjustments do you make? 

107. Adapted with permission from: Hudesman, J. & Crosby, S. (2014). Self- regulated learning (SRL) 
student workbook. Prepared for Essex County College. Unpublished manuscript. Contact: lohn. 
Hudesman@ret.citvtech.cunv.edu . See also: Finkelman, J., Hudesman,]., Flugman, B., & Crosby, 
S. (2014). An enhanced formative assessment and self-regulated learning program: From the 
classroom to the workplace. Awstm Journal of Psychiatry and Behavioral Science, /(I), 1002-1009. 

108. Quiz adapted with permission from: Finkelman,]., Hudesman,]., Flugman, B., & Crosby, S. (2014). 
An enhanced formative assessment and self-regulated learning program: From the classroom to 
the workplace. Austin Journal of Psychiatry and Behavioral Science, /(I), 1002-1009. First cited 
in: Zimmerman, B. J., Moylan, A., Hudesman, J., White, N., & Flugman, B. (201 1). Enhancing self- 
reflection and mathematics achievement of at-risk urban technical college students. Psychological 
Test and Measurement Modeling, 53(1), 141-160. 

109. Sample Quiz Reflection adapted with permission from: Finkelman,]., Hudesman,]., Flugman, 
B., & Crosby, S. (2014). An enhanced formative assessment and self-regulated learning program: 
From the classroom to the workplace. Austin journal of Psychiatry and Behavioral Science, 1(1), 
1002-1009. First cited in: Zimmerman, B. J., Moylan, A., Hudesman, J., White, N., & Flugman, B. 
(2011). Enhancing self-reflection and mathematics achievement of at-risk urban technical college 
students. Psychological Test and Measurement Modeling, 53(1), 141-160. 

110. Reprinted with permission from: Hudesman,]., & Crosby, S. (2014). Self- regulated learning (SRL) 
student workbook. Prepared for Essex County College. Unpublished manuscript, p. 23. Contact: 
lohn.Hudesman@ret.citvtech.cunv.edu . 

111. Reprinted with permission from: Hudesman,]., & Crosby, S. (2014). Self- regulated learning (SRL) 
student workbook. Prepared for Essex County College. Unpublished manuscript, p. 73. Contact: 
lohn.Hudesman@ret.citvtech.cunv.edu . 

112. Reprinted with permission from: Hudesman,]., & Crosby, S. (2014). Self- regulated learning (SRL) 
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